Method and apparatus for rapid approximation of system model

ABSTRACT

Systems, methods, apparatus and mechanisms that iteratively generate one or more equations (conforming to one or more functions or functions types) of increasing complexity (e.g., degree or order) to provide one or more corresponding approximations of a System Under Study (SUS), where each approximation or “fit” provides additional information about the SUS and where the various equations may be summed to provide a final “fit” having an adequate level of accuracy.

FIELD OF THE INVENTION

The invention relates generally to mechanisms for rapidly generating amathematical model to closely approximate a system under study (SUS).

BACKGROUND

Traditional approaches to calculate a summary statistic (mean, variance,etc.) is to generate a large number of random input vectors (i.e., setsof input parameters) and evaluate the corresponding output vectors(i.e., sets of output parameters) by conducting tests on the SystemUnder Study SUS. This is commonly known as Monte Carlo simulation orMonte Carlo experimentation. The term SUS is used in a relatively broadmanner to refer to a mechanical system, electrical system, chemicalsystem, biological system, electromechanical system, software system,financial system and so on. Mean, probability of occurrence of certainevents, variance, and other summary statistics of the output vectors aretypically computed.

Traditional system modeling and simulation methods are extremely costlyin terms of computational, memory, input/output and other resources,hardware experimental cost, in terms of time and so on due to largenumbers of input and operational variables, many interdependenciesbetween the variables and so on.

A system may comprise an article of manufacture having a plurality ofassemblies, where each of the assemblies comprises a plurality ofcomponents. For example, complex models and simulations for safetysystems such as airbags, seat belts and the like are created to ensurethat these critical systems will perform their intended functions undervarious operating conditions. This modeling and simulation of safetysystems (or other systems) is performed prior to production of suchsystems or integration of such systems into a product. Key variablesthat contribute to uncertainty include passenger weight (e.g., 120-200lbs), passenger height (e.g., 56-80 inches), lower seat position (e.g.,14-28 inches), recline angle (e.g., 0-27 degrees), manufacturingvariability (e.g., change in a critical dimension by +/−0.002 inches)and so on.

A system may comprise a service associated with a plurality of subservices and/or service components. For example, in a financial model toevaluate foreign corporate bond investment, there are multipleparameters that can vary and create uncertainty in terms ofdecision-making. Key uncertainty variables might include local borrowingrate, local market yield for similar bonds, foreign exchange rates,local tax rate, foreign tax rate and so on. The financial model itselfmay be included within a larger risk model associated with an entityowning a plurality of investments including foreign corporate bonds. Amodeling and simulation of the financial model must be performed priorto investment.

SUMMARY

Various embodiments provide systems, methods, apparatus and/ormechanisms that iteratively generate one or more equations (conformingto one or more functions or function types) of increasing complexity(e.g., degree or order) to provide one or more correspondingapproximations of a System Under Study (SUS), where each approximationor “fit” provides additional information about the SUS, where thevarious equations may be summed to provide a final “fit” having anadequate level of accuracy. Models defined according to polynomials,orthogonal polynomials and/or other functions may be iterativelyimproved using deterministic data/techniques in which only thosevariables and model terms determined at each iteration to be dominantcontributors to model accuracy are further processed. Analyticalintegration techniques may be used to estimate mean, variance and othersummary statistics. In this manner, the various costs associated withprocessing non-dominant contributors are avoided while achieving asufficiently high level of accuracy.

In one embodiment, an apparatus is provided for modeling a System UnderStudy (SUS), the SUS exhibiting at least one output adapted in responseto a plurality of input parameters, the apparatus comprising a processorconfigured to: generate an improved approximation model for said SUScomprising a summation of an initial approximation model, a constantterm, and a plurality of polynomials, each polynomial being associatedwith a respective N-tuplet of input parameters for greater than zerointeger values of N, wherein polynomials included within said improvedapproximation model are generated by iteratively performing thefollowing steps until a sufficiently accurate fit is achieved: (1) rankordering the N-tuplets of input parameters; (2) selecting a highestranking N-tuplet of input parameters from among the remaining N-tupletsof input parameters to be processed; (3) fitting a polynomial involvinginput parameters in the selected N-tuplet of input parameters; (4)updating the current approximation model by adding the fittedpolynomial; (5) adjusting the constant term as needed to match theevaluations of the SUS and the current approximation model at a point;and (6) evaluating a difference between the SUS and the currentapproximation model to determine if a sufficiently accurate fit isachieved.

BRIEF DESCRIPTION OF THE DRAWINGS

The teachings herein can be readily understood by considering thefollowing detailed description in conjunction with the accompanyingdrawings, in which:

FIG. 1 depicts a high-level block diagram of a system suitable for usein implementing the various embodiments;

FIG. 2 depicts a flow diagram of a method according to variousembodiments;

FIGS. 3A-3E depict flow diagrams of various SUS model fitting methods;

FIGS. 4A-4B depict flow diagrams of a method of fitting a first fitapproximation model (1FA) to a SUS according to various embodiments;

FIG. 5 depicts a flow diagram of a method of generating a first fitpolynomial model for a selected input parameter associated with a SUS;

FIGS. 6A-6C graphically depict n-point distributions useful inunderstanding the various embodiments;

FIGS. 7A-7B depict a flow diagrams of a method of generating a secondfit approximation model (2FA) involving pairs of input parametersassociated with a SUS;

FIG. 8 depicts a flow diagram of a method of generating a 2-parameterpolynomial for a selected input parameter pair according to oneembodiment; FIGS. 9A-9B graphically depict two-dimensional n-pointdistributions useful in understanding the various embodiments;

FIG. 10 depicts a flow diagram of a method of processing two outputparameters according to various embodiments;

FIG. 11 graphically depicts a flow diagram of a method of generatingstress points, generating corresponding input vectors, and evaluating aSUS and/or SUS-characterizing mathematical model according to oneembodiment;

FIG. 12 describes method of post-processing SUS-characterizinginformation according to one embodiment; and

FIG. 13 depicts a high-level block diagram of a computing devicesuitable for use in performing functions described herein with respectto the various embodiments.

To facilitate understanding, identical reference numerals have beenused, where possible, to designate identical elements that are common tothe figures.

DETAILED DESCRIPTION OF THE INVENTION

Systems, methods, apparatus, mechanisms and the like for efficientlymodeling a System Under Study (SUS) or component(s) thereof are depictedand described herein. Generally speaking, the various embodimentsdescribed herein implement a Rapid Accurate Estimator Machine (RACE)adapted to efficiently develop and validate a mathematical model closelyapproximating a behavior of a System Under Study (SUS) within specifiedregion of interest. The RACE is also adapted to efficiently andaccurately determine mean, variance, probability of occurrence of anevent, and other summary statistics associated with various outputs andfunctions of outputs of the SUS for given probability distributions ofthe SUS input parameters.

In various embodiments, the SUS is presented by a user or SUS-source orSUS-characterizer in the form of a computer simulation, a softwaresystem, a physical system, a chemical system, a biological system, afinancial system or any other simple/complex system (or combinationthereof) adapted to receive input data such as test vectors which, whenprocessed by the SUS, provide output data or results that may becomputed or observed.

Various embodiments develop a mathematical model to closely approximatethe behavior of a SUS within a specified region of interest. Developingaccurate mathematical models to represent SUS is a well-known challengeand can require significant test or simulation resources. In addition tosummary statistics computation, the invention provides a systematicprocess for developing and validating a mathematical model.

Generally speaking, the various embodiments provide a mechanism forgenerating an accurate model of a System Under Study (SUS) in a highlyefficient manner to avoid executing the thousands or millions ofrandom/systematic simulations typically necessary to generate anaccurate SUS model and/or its summary statistics.

The various embodiments enable rapid decision making by those takingaction in response to the SUS behavior or performance. In this manner,time critical applications such as high-speed trading may be improvedwith significantly less computing resources or without any significantincrease in applied computing resources.

Various embodiments of this invention utilize polynomials (e.g.orthogonal polynomials) or other classes of functions (e.g. Fourierseries, wavelets, Walsh functions) which are iteratively fit to SUSevaluations. In the various embodiments, polynomials are fitsuccessively starting with single parameter terms and progressing to,when applicable, multiple input parameter terms. In various embodiments,input parameters and interactions of multiple input parameters to be fitare prioritized based on structured test vectors (e.g. based onorthogonal arrays) executed on the SUS.

Various embodiments allow for optimization of polynomial fits to onlyinclude dominant contributing polynomial terms in the SUS-characterizingmathematical model (SCMM) to improve processing performance whilemaintaining acceptable accuracy. In various embodiments, the iterativemathematical model building process is optimized to stop adding newterms when accuracy requirements are attained. Various embodiments ofthis invention prescribe use of structured test vectors (e.g. based onorthogonal arrays) for assessing “goodness” of fit of aSUS-characterizing mathematical model versus actual SUS evaluationsusing goodness of fit metrics (e.g. a maximum absolute difference, a sumof absolute differences, a mean square of the difference and/or a meansquare of the selected top absolute differences, a mean square error,other criteria, etc.).

The SUS may comprise any system, such as a system with at least oneoutput (including one or more outputs or characteristics of interest toa user or other entity) adapted in response to a plurality of inputs.Generally speaking, the various methods iteratively generate one or moreequations (conforming to one or more functions or function types) ofincreasing complexity (e.g., degree or order) to provide one or morecorresponding approximations of the SUS, where each approximation or“fit” provides additional information about the SUS, wherein the variousequations may be summed to provide a final “fit” having an adequatelevel of accuracy.

Various embodiments of this invention utilize analytical and numericalintegration techniques to compute summary statistics (e.g. mean,variance, etc.) from the generated SUS-characterizing mathematical modelSCMM.

FIG. 1 depicts a high-level block diagram of a system suitable for usein implementing the various embodiments. The system 100 of FIG. 1comprises one or more data processing elements, computing devices,network elements and the like cooperating as described herein toimplement various embodiments. Not all of the described data processingelements, computing devices, network elements and the like are necessaryto implement each embodiment. The exemplary system 100 described hereinis provided for illustrative purposes only.

The system 100 of FIG. 1 contemplates a server, workstation or othercomputing device implementing a Rapid Accurate Estimator Machine (RACE)105 operating in accordance to the various embodiments, such asdescribed herein and with respect to the various other figures.

In various embodiments, the RACE 105 operates with process descriptorsand/or other information associated with a System Under Study (SUS) orportion thereof to generate a sufficiently accurate model of the SUS orportion thereof such that operation of the SUS may be visualized,explored and otherwise understood. An SUS descriptor generally comprisesdescriptive information associated with the SUS suitable for use inmodeling SUS behavior, confirming results of SUS behavior models and thelike. The SUS descriptor may be of any format suitable for processing bythe RACE 105.

A sufficiently accurate fit of a model or component thereof occurs when,illustratively, an evaluation of the model or component thereof and therelevant SUS (or component thereof) or other SUS portions at stresspoints results in a difference below a threshold level. The differencethreshold level may be defined as a number of relevant units (e.g., 1,10 etc.), a percentage of accuracy (e.g., 80%, 99.99% etc.) a number ofstandard deviations (e.g., 3SD, 6SD etc.) or some other representationand/or units appropriate to the particular SUS. Generally speaking, anytechnique for defining “good fit” may be used within the context of thevarious embodiments.

In various embodiments, the generated model is used to drive furtheractions, such as computation of mean and standard deviation, as SUSoperational dynamics and the like become more accurately characterizedby the RACE 105.

In various embodiments, the system 100 may further include one or bothof a system definition server 106 and an action server 107, either ofwhich may be in communication with the RACE 105 and/or each other via adirect communication link or via a network 108. The system definitionserver 106 and action server 107 may be implemented together orseparately, in a single computing device or via multiple computingdevices. In various embodiments, a single server, workstation or othercomputing device is adapted to implement the various functionsassociated with the RACE 105, system definition server 106 and actionserver 107.

The optional system definition server 106 may comprise a standard serveror other computing device adapted to provide to the RACE 105 SUSdescriptors and/or other information suitable for characterizinginput/output functions, transfer functions, exceptions and/or othercharacterizing information associated with a SUS. This information maybe provided to the RACE 105 according to any suitable format.

The optional action server 107 receives SUS model data, SUSvisualization data and other data from the RACE 105 to perform variousactions in response thereto. Such actions may include automatedprocesses such as automated stock trading, automated failure analysis,automated production or assembly adjustments and so on. Generallyspeaking, the action server 107 operates to take action in response toone or more SUS models generated by the RACE 105, and/or different typesof data derived by the RACE 105 during and/or after the generation ofsuch one or more SUS models. In various embodiments, the action server107 may also receive and use SUS descriptors or other SUS-representativeinformation provided by, illustratively, the optional the systemdefinition server 106.

As shown in FIG. 1, the RACE 105 is configured in a particular manner interms of hardware, software, input/output resources, connecteddevices/functions and the like. However, it will be appreciated by thoseskilled in the art that the RACE 105 may be configured according to anyone of a number of computing topologies or configurations. That is, theRACE 105 may comprise a general purpose computer, a special purposecomputer, a specific type of server and/or any other computing devicecapable of performing the various functions described herein. Thus, theRACE 105 as described herein with respect to FIG. 1 may also beimplemented as a general purpose computing device, such as described inmore detail below with respect to FIG. 13.

As depicted in FIG. 1, the RACE 105 includes one or more processors 110,a memory 120, communications interfaces 130 and an input-output (I/O)interface 140. The processor 110 is coupled to each of memory 120,communication interfaces 130, and I/O interface 140.

The processor 110 is configured for controlling the operation of RACE105, including operations supporting the methodologies described hereinwith respect to the various figures, any interactions with the systemdefinition server 106, action server 107, network 108 or otherdevices/systems. The memory 120 is configured for storing informationsuitable for use in generating a sufficiently accurate model of a SystemUnder Study (SUS) according to the various embodiments. Specifically,memory 120 may store programs 121, data 122 and so on. Within thecontext of the various embodiments, the programs 121 and data 122 mayvary depending upon the specific functions implemented by the RACE 105.For example, as depicted in FIG. 1, the programs portion 121 of memory120 includes functional modules denoted as follows: an Entry Engine121-EE, a Tuplet and Vector Engine 121-TVE, a Human Simulation Softwareand Hardware Interface 121-HSSHI, an Equation Engine 121-EQE, a SummaryEngine 121-SE and a Visualization Engine 121-VE. Similarly, the datastorage portion 122 of memory 120 is depicted as including model data122-MD and engine data 122-ED. The functional blocks 121-EE, 121-TVE,121-SE, 121-HSSHI, 121-EQE, and 121-VE may also be referred as EE, TVE,SE, HSSHI, EQE, and VE, respectively.

The Entry Engine EE provides a mechanism to receive and store Userand/or System inputs associated with the RACE methodologies, such asinformation received from a User, a System definition server 106, a SUSor some other source.

The Tuplet Vector Engine TVE provides a mechanism to generate inputvectors used for evaluating the SUS, which could be present in the formof hardware, simulation software, or some other mechanism. The TVE mayalso be used for evaluation of SUS-characterizing mathematical models.

The Summary Engine SE provides a mechanism for computing summarystatistics such as mean, variance and the like such as derived from theSUS-characterizing mathematical model.

The Human Simulation Software and Hardware Interface HSSHI provides amechanism for transmitting input parameter data to SUS, receiving outputdata from SUS and generally interacting with human operators, computersimulations, hardware in loop systems, embedded software systems, andother interfaces to execute and collect data from the SUS and/orSUS-characterizing mathematical model.

The Equation Engine EQE provides a mechanism for processing SUSevaluations data to generate, refine or otherwise update theSUS-characterizing mathematical model to iteratively improve accuracysuch that the SUS-characterizing mathematical model more closelyapproximates actual behavior of the SUS. It also provides a mechanismfor assessing “goodness” (i.e., sufficiency) of fit of theSUS-characterizing mathematical model.

The Visualization Engine VE provides a mechanism for processing SUSevaluations and SUS-characterizing mathematical model evaluations toprovide output information, such as for User and/or System consumptionin an appropriate format, such as via databases, metadata, tables,graphs, charts, and other data storage or data visualization techniques.

Generally speaking, the memory 120 may store any information suitablefor use by the RACE 105 in implementing one or more of the variousmethodologies or mechanisms described herein. It will be noted thatwhile various functions are associated with specific programs ordatabases, there is no requirement that such functions be associated inthe specific manner. Thus, any implementations achieving the functionsof the various embodiments may be used.

The communications interfaces 130 may include one or more servicessignaling interfaces such as a Wi-Fi or WiMAX interface, a 3G wirelessinterface, a 4G wireless interface, an Ethernet interface and the likefor supporting data/services signaling between RACE 105 and the network108. It will be appreciated that fewer or more, as well as different,communications interfaces may be supported. The various communicationsinterfaces 130 are adapted to facilitate the transfer of files, datastructures, messages, requests and the like between various entities inaccordance with the embodiments discussed herein.

The I/O interface 140 may be coupled to one or more presentation devices(PDs) 102 such as associated with display devices for presentinginformation to a user, one or more input devices (IDs) 103 such as touchscreen or keypad input devices for enabling user input, and/orinterfaces enabling communication between the RACE 105 and othercomputing, networking, presentation or input/output devices (not shown).

Presentation devices 102 may include a display screen, a projector, aprinter, one or more speakers, and the like, which may be used fordisplaying data, displaying video, playing audio, and the like, as wellas various combinations thereof. The typical presentation interfacesassociated with user devices, including the design and operation of suchinterfaces, will be understood by one skilled in the art.

Input devices (ID) 103 may include any user control devices suitable foruse in enabling a local or remote user of the RACE 105 to interact withthe RACE 105. For example, the input devices 103 may include touchscreen based user controls, stylus-based user controls, a keyboardand/or mouse, voice-based user controls, and the like, as well asvarious combinations thereof. The typical user control interfaces ofuser devices, including the design and operation of such interfaces,will be understood by one skilled in the art.

Although primarily depicted and described as having specific types andarrangements of components, it will be appreciated that any othersuitable types and/or arrangements of components may be used for RACE105.

It will be appreciated that the functions depicted and described hereinmay be implemented in software and/or hardware, e.g., using a generalpurpose computer, one or more application specific integrated circuits(ASIC), and/or any other hardware equivalents. In one embodiment, thevarious programs depicted as loaded within memory 120 are executed bythe processor(s) 110 to implement their respective functions. It willalso be appreciated that the various programs may be stored on acomputer readable storage medium prior to being loaded into memory 120;such computer readable storage media comprising semiconductor memorydevices, magnetic media, optical media, electromagnetic media and thelike. Generally speaking, any form of tangible computer memory may beused to store computer instructions which, when executed by theprocessor 110, operate to perform the various methods and functionsdescribed herein.

It is contemplated that some of the steps discussed herein as softwaremethods may be implemented within hardware, for example, as circuitrythat cooperates with the processor to perform various method steps.Portions of the functions/elements described herein may be implementedas a computer program product wherein computer instructions, whenprocessed by a computer, adapt the operation of the computer such thatthe methods and/or techniques described herein are invoked or otherwiseprovided. Instructions for invoking the inventive methods may be storedin tangible fixed or removable media, transmitted via a data stream in abroadcast or other tangible signal-bearing medium, and/or stored withina memory within a computing device operating according to theinstructions.

FIG. 2 depicts a flow diagram of a method according to variousembodiments. Specifically, the method 200 of FIG. 2 depicts severalhigh-level processing steps performed by an apparatus operative toiteratively improve a SUS-characterizing mathematical model SCMM to asufficient level of accuracy, such as the RACE 105 described above withrespect to FIG. 1. Various steps described herein with respect to FIG. 2will be described in more detail below with respect to further figures.

At step 210, the method 200 receives input information from user,software module, interface and/or system input means. For example, theinput information may be received via the Entry Engine 121-EE associatedwith the RACE 105. The input information may comprise, illustratively,information such as definitions of SUS input parameters and outputparameters, SUS input/output parameter distributions or ranges ofinterest if applicable, joint distributions of input parameters ifapplicable, SUS type, any interface requirements and/or otherinformation pertaining to the SUS or entities in communication with theRACE 105.

The term “User” as described herein may comprise a human operator at aworkstation or other computing device, a computing machine, softwareexecuted upon a computing machine, an external system configured toprovide inputs to and process outputs from the RACE 105 and/or SUS andso on.

The term “System” as described herein may comprise any internal ordefault operations, metadata, configuration files or settings configuredto provide inputs to and process outputs from the RACE 105 and/or SUSand so on.

Input parameter distribution, if applicable, can be provided asGaussian, Log Normal, uniform, triangular, and/or other distributionfunction.

The term “input parameters” may also be referred to herein as “factors”or “input factors” while the term “output parameters” may also bereferred to herein as “outputs” or “responses” such as in accordancewith the appropriate terminology associated with a particular SUS orSUS-characterizing mathematical model SCMM. For example, in a financialmodeling SUS, input parameters may comprise an interest rate, anexchange rate, a volatility indicator and the like. Further, the Usermay supply a mean and variance for the input parameters.

The term “region of interest” RI of an input parameter as describedherein may be defined as a nominal operating range of values, a highestimpact range of values, and/or a selected range of values. In variousembodiments, the region of interest RI is the region spanned by the meanplus/minus a certain multiple of the standard deviation for the inputparameter (e.g., 2, 3 or other multiple). In various embodiments, aregion of interest for an input parameter may be defined with respect toa specified percent of probability (80%, 90% or other). If a range ofinterest (e.g., upper and lower bound) of an input parameter isspecified, then the range of interest may also be used to define aregion of interest associated with that input parameter. The term“dimensional limit” DL as described herein is defined as the number ofinput parameters specified by the User or System (e.g., where a Userand/or System has specified 10 input parameters, the DL is 10).

In various embodiments, the region of interest RI for the collection ofinput parameters is the multidimensional region spanned by the region ofinterest for all input parameters. A Center Point CP is a point in theDL dimensional RI of the input parameter space. In some embodiments theCenter Point CP coincides with the centroid of the DL dimensional RI ofthe input parameter space. In another embodiment, the CP would belocated at the modes of the probability distributions of the inputparameters. In some embodiments, the CP would be located at the mean ofthe probability distributions of the input parameters, or other. In someembodiments, CP is a point specified by the User, System or default as asuitable point in RI.

At step 220, the method 200 generates a simplified model to provide agood fit (close approximation) of the SUS in the RI using a systematicalgorithmic process. Specifically, SUS information as well as variousConstraints may be used within the context of approximating the SUS bygenerating a SUS-characterizing mathematical model SCMM to closelyapproximate the SUS in a specific region of interest. For example, theEquation Engine 121-EQE associated with the RACE 105 may be used forthis purpose.

Generally speaking, at step 220 various processing steps are performedto generate and iteratively improve a SUS-characterizing mathematicalmodel SCMM until a sufficient degree of model accuracy is attained withrespect to one or more response variables of interest (e.g., risk level,output voltage of a circuit, stress in a component, failure rate,quality level, responsiveness, cost of materials, cost of ownership andso on). User or System specified Constraints may include accuracyspecifications, goodness of fit criteria, financial limitations,resource limitations, mode of operation, loop back criteria, polynomialorders for various passes, and other. Constraints may be specifiedthrough the EE. Mode of operation may include specifications such asprocessor percentage utilization, resource consumption rates, settingsto increase output speed, and other parameters. The various functionsassociated with step 220 will be described in more detail below withrespect to FIGS. 3A-3E. The finalized model from step 220 is alsoreferred to as a final fit SUS-characterizing mathematical model SCMM.The working model, before it is finalized is also referred to as aCurrent SUS-characterizing mathematical model CSCMM. The CSCMM may alsobe referred as a current approximation model.

At step 230, the method 200 computes summary statistics, accuracyestimates and savings ratio associated with a final fitSUS-characterizing mathematical model, such as mean, variance, accuracyestimates, savings ratios, parameter effects, computable statistics, andother information. For example, the Summary Engine 121-SE associatedwith the RACE 105 may be used for this purpose.

Generally speaking, at step 230 the method 200 stores/provides afinalized SCMM, computes summary statistics, transmits any processupdate messages or alarms, computes effects of input parameters or otherparameters, computes interaction effects of input parameters or otherparameters, computes savings ratio and/or performs other post processingcomputation or function as appropriate.

The model for closely approximating the SUS in the RI is finalized whengoodness of fit is within User, System, and/or default specifications,DL limits are met, and/or Constraints restrict further improving of themodel.

Summary statistics may include mean, standard deviation, variance,confidence intervals, and others. At step 230, given the equation forthe SUS-characterizing mathematical model, these statistics are computedby using formulae for moments of the input parameter distributions,analytical integration, or through numerical integration or othermethods. Correlations, if any, among input parameters may becharacterized by the joint distribution of the input parameters, whichis utilized in some embodiments to compute mean, standard deviation,variance, confidence intervals, and the like.

Computations may include input parameter effects and effects ofinteractions among the input parameters for each output parameter. Ifthere are multiple output parameters, there would be an equation foreach output parameter and the process is same for each output parameter.Additionally, in the case of multiple output parameters, SE may computecorrelations among the output parameters in addition to otherstatistics. Provided an equation as developed using the processesdescribed herein, input parameter effects, input parameter interactionsfor single and multiple outputs and correlations of output parametersmay be calculated using standard statistical processes.

The savings ratio SR is a ratio of the number of evaluations using theRACE 105 versus the number of Monte Carlo evaluations required to getsimilarly accurate mean and variance calculations. In the case ofmultiple outputs, savings ratio SR may be computed for each outputparameter.

At step 240, the various results of the prior steps are displayed and/orstored in a suitable manner for later use by a User, System and thelike. For example, the Visualization Engine 121-VE associated with theRACE 105 may be used for this purpose. Generally speaking, at step 240the method 200 stores/provides summary statistics in formats for Userand System consumption, displays summary statistics and parametereffects in graphical formats, displays results via graphics or otherpresentation information and/or performs or provides other visualizationinformation as appropriate and/or other post-processing.

For example, the VE may store the summary statistics, parameter effects,and other computations from step 230 in formats for consumption by Userand/or System. These formats include tables, graphs, databases, andother formats specifically configured for the specific User or System.If User or System specified multiple output parameters, VE stores datafor consumption for each output parameter.

The VE may display the summary statistics, parameter effects, and othercomputations from the various embodiments described herein. Relevantgraphical formats may include bell curves, probability distributions,input parameter effects graphs, tables, scatter diagrams, histograms,response surface graphs, interaction effects, and other formats. If Useror System specified multiple output parameters, VE displays the resultsfor each output and combinations thereof.

FIGS. 3A-3E depict flow diagrams of various SUS model fitting methodssuitable for use in, illustratively, implementing step 220 of the method200 of FIG. 2.

FIG. 3A depicts a method 300A according to one embodiment forimplementing step 220 of the method 200 of FIG. 2.

At step 310, the method 300A (e.g., via the TVE) generates vectorscomprising input parameters within a region of interest RI, representinga plurality of stress points SP in the region of interest RI, andevaluates the SUS at the selected stress points SP. An exemplary methodof generating vectors, selecting stress points and evaluating the SUS atthe selected stress points suitable for use in implementing step 310will be described in more detail below with respect to FIG. 11.

Generally speaking, the SP may be generated to expose the inability of amodel to adequately match the SUS output. In various embodiments, SP maybe generated using an orthogonal array. In other embodiments, SP may beselected using fractional factorial designs, full factorial designs,design of experiments, space filling designs, and/or other methods. Invarious embodiments, SP selection may be dependent on User input, Systeminput and/or specified Constraints. In various embodiments, the SUS isevaluated at the SP via, illustratively, the HSSHI.

As an example, in an embodiment having four input parameters (labeled u,v, w, and x) the process may select two levels or values for each of theinput parameters. The levels would be labeled as u₁ and u₂ for parameteru, v₁ and v₂ for parameter v, and so on. It should be noted that thevalues u₁ and u₂ may be chosen within the RI for the parameter u; thevalues v₁ and v₂ may be chosen within the RI for the parameter v; and soon.

In operation, the various embodiments may select the orthogonal array L8depicted below in Table 1 for generating stress points. The array haseight rows and seven columns. The method may, illustratively, select thefirst four columns to generate the stress points. The correspondingeight stress points SP are depicted in Table 2. The eight rows of thistable represent the eight vectors for the eight stress points. Eachvector has four components corresponding to the four columns labeledParameter u, Parameter v, Parameter w, and Parameter x in Table 2.

TABLE 1 Col1 Col2 Col3 Col4 Col5 Col6 Col7 1 1 1 1 1 1 1 1 2 1 1 1 2 2 22 3 1 2 2 1 1 2 2 4 1 2 2 2 2 1 1 5 2 1 2 1 2 1 2 6 2 1 2 2 1 2 1 7 2 21 1 2 2 1 8 2 2 1 2 1 1 2

TABLE 2 Parameter Parameter Parameter Parameter u v w x 1 u₁ v₁ w₁ x₁ 2u₁ v₁ w₁ x₂ 3 u₁ v₂ w₂ x₁ 4 u₁ v₂ w₂ x₂ 5 u₂ v₁ w₂ x₁ 6 u₂ v₁ w₂ x₂ 7 u₂v₂ w₁ x₁ 8 u₂ v₂ w₁ x₂

In another example, again with same four input parameters, the variousembodiments may select a full factorial design consisting of allcombinations of the two levels each for the four parameters. Thecorresponding 16 stress points are depicted below with respect to Table3.

TABLE 3 Parameter Parameter Parameter Parameter u v w x 1 u₁ v₁ w₁ x₁ 2u₁ v₁ w₁ x₂ 3 u₁ v₁ w₂ x₁ 4 u₁ v₁ w₂ x₂ 5 u₁ v₂ w₁ x₁ 6 u₁ v₂ w₁ x₂ 7 u₁v₂ w₂ x₁ 8 u₁ v₂ w₂ x₂ 9 u₂ v₁ w₁ x₁ 10 u₂ v₁ w₁ x₂ 11 u₂ v₁ w₂ x₁ 12 u₂v₁ w₂ x₂ 13 u₂ v₂ w₁ x₁ 14 u₂ v₂ w₁ x₂ 15 u₂ v₂ w₂ x₁ 16 u₂ v₂ w₂ x₂

At step 310, the method 300A (e.g., via the EQE) also initializes thecurrent SUS-characterizing mathematical model CSCMM to zero or to aninitial User or System provided approximation model, if any.

At step 320, the method 300A (e.g., via the EQE) improves the CSCMM(current SUS model) by adding single parameter terms and adjusting theconstant. That is, the method 300A at step 320 generates a first fitapproximation model for SUS, model comprising of the CSCMM and singleparameter polynomials of adequate orders for input parameters associatedwith the SUS, so as to closely approximate the SUS in the RI. Theresulting model, which comprises a sum of the CSCMM, single parameterpolynomials, and a constant term, is referred to as the 1FA model. Anexemplary method of fitting a 1FA model suitable for use in implementingstep 320 will be described in more detail below with respect to FIGS.4A-4B.

Within the context of a finance SUS example, the 1FA model may comprise,illustratively, a constant term, a polynomial for interest rate ofappropriate order, a polynomial for the exchange rate of appropriateorder, and a polynomial for each of the other input parameters ofappropriate orders. Other modifications/models are also contemplated bythe inventors.

At step 330, a determination is made as to whether the evaluation of the1FA model at the SP indicates that the 1FA model represents a good fitto the SUS. If the 1FA model represents a good fit, CSCMM is set equalto the 1FA model and the method 300A exits at step 390. If the 1FA modeldoes not represent a good fit, CSCMM is set equal to the 1FA model andthe method 300A proceeds to step 340.

Generally speaking, differences between the 1FA model (or anotherSUS-characterizing mathematical model) and SUS evaluations at thecorresponding stress points are assessed according to goodness of fitcriteria such as a maximum absolute difference, a sum of absolutedifferences, a mean square of selected top absolute differences, a meansquare error, or other. The goodness of fit criteria may be specified byUser or System and could be a part of inputs supplied, default settings,or Constraints. If the difference between the SUS and the 1FA model (orother SUS-characterizing mathematical model) according to the goodnessof fit criteria is outside User or System specification, then the fit isconsidered not a good fit, CSCMM is set equal to the 1FA model, and theCSCMM is broadened in scope to improve its accuracy by proceeding tostep 340. If the difference is within the User or System specification,then the fit is considered a good fit, CSCMM is set equal to the 1FAmodel, and the method 300A exits at step 390.

At step 340, the method 300A (e.g., via the EQE) improves the CSCMM(current SUS model) by adding two parameter terms and adjusting theconstant. That is, at step 340, 2-parameter terms of the type x₁ ^(m)x₂″ are systematically added to the CSCMM to provide thereby a secondfit approximation model, where m and n are positive integers greaterthan zero and x₁ and x₂ are input parameters. The resulting model, whichcomprises a sum of the CSCMM, two parameter terms, and a constant term,is referred to as the 2FA model. An exemplary method of fitting a 2FAmodel suitable for use in implementing step 340 will be described inmore detail below with respect to FIGS. 7A-7B.

In the finance example, the 2FA model may include 2-parameter termsinvolving interest rate and exchange rate of appropriate orders, termsinvolving interest rate and volatility of appropriate orders, termsinvolving exchange rate and volatility of appropriate orders, as well as2-parameter terms of other combinations of input parameters ofappropriate orders.

At step 350, the 2FA model is evaluated at the previously selectedstress points to determine whether the 2FA model represents a good fitto the SUS, such as described above with respect to the assessment ofthe 1FA model at step 330. If the 2FA model represents a good fit, thenthe CSCMM is set equal to the 2FA model and the method 300A exits atstep 390. If the 2FA model does not represent a good fit, then the CSCMMis set equal to the 2FA model and the method 300A proceeds to step 360.

At step 360, the method 300A (e.g., via the EQE) improves the CSCMM(current SUS model) by adding N parameter terms and adjusting theconstant. That is, at step 360, N-factor terms, where N is initially 3,of type x₁ ^(m) x₂ ^(n) x₃ ^(o) are systematically added to the CSCMM byconsidering triplets of input parameters to provide thereby a third fitapproximation. Here m, n, and o are positive integers greater than zeroand x₁, x₂, and x₃ are input parameters. The resulting model, whichcomprises a sum of the CSCMM, 3 parameter terms, and a constant term, isreferred to as the 3FA model.

In the finance example, the 3FA model may include 3-factor termsinvolving interest rate, exchange rate, and volatility of appropriateorders as well as 3-factor terms of other combinations of three inputparameters of appropriate orders.

At step 370, the 3FA model is evaluated at the previously selectedstress points to determine whether the 3FA model represents a good fitto the SUS, such as described above with respect to the assessment ofthe 1FA model at step 330. If the 3FA model represents a good fit, thenCSCMM is set equal to the 3FA model and the method 300A exits at step390. If the 3FA model does not represent a good fit, then CSCMM is setequal to the 3FA model and the method 300A proceeds to step 380.

At step 380, the variable N is incremented and steps 360-370 arerepeated. In particular, for each repetition of step 360, higher levelpolynomials may be added such as previously described to adapt the NFAmodel such that it is improved in terms of fit. That is, the loopprocess of steps 360-380 systematically increases the number of inputparameters in a term added to the CSCMM by one to improve accuracy. Ifthe resulting goodness of fit of the NFA model evaluated at step 370 isnot good, then CSCMM is set equal to the NFA model and N is incrementedat step 380, and steps 360 and 370 may be repeated. If at step 370, theNFA model represents a good fit, then CSCMM is set equal to the NFAmodel and the method 300A exits at step 390.

It is noted that the term NFA model implies that the model containspolynomial terms involving N parameters. For example, if N is 6, theterms being considered in the NFA analysis would be of type x₁ ^(m) x₂^(n) x₃ ^(o)x₄ ^(p)x₅ ^(q)x₆ ^(r). Here m, n, o, p, q, and r arepositive integers greater than zero and x₁, x₂, x₃, x₄, x₅, and x₆ areinput parameters. For this example, the analysis would be called 6FAanalysis and the resulting model which includes the 6-factor terms wouldbe called the 6FA model.

In various embodiments, steps 360-370 are repeated only for apredetermined number of iterations, until a budget is exhausted, oruntil some other threshold event occurs.

Generally speaking, the limit of N may be selected as the dimensionallimit DL of the user inputs. For example, if the User or System providesonly 5 inputs, the DL is 5 and 5FA will be the maximum that RACE 105 canprocess. In another example, the number of input parameters could be 2in which case the DL is 2 and 2FA will be the maximum that RACE 105 canprocess. In various embodiments, when the method 300 reaches the DL,then message is transmitted to the User or System indicative of thisevent to trigger thereby additional processes or interaction by the Useror System.

At step 390, the SUS-characterizing mathematical model SCMM is set tothe latest approximation model (CSCMM) from the preceding step andmethod 300A exits. Optionally, the SCMM is also simplified at step 390.

In various embodiments, the terminology N-tuplet of input parameters,where N is an integer greater than 0, is used to refer to a group of Ninput parameters. When N=1, N-tuplet refers to a single input parameter;when N=2, N-tuplet refers to a pair of input parameters; when N=3,N-tuplet refers to a triplet of input parameters; and so on.

In various embodiments, the terminology N-dimensional space, where N isan integer greater than 0, is used while working with a NFA process.When N=1, N-dimensional space stands for a straight line. When N=2,N-dimensional space stands for a plane. When N=3, N-dimensional spacerefers to a three dimensional space. When N>3, N-dimensional spacestands for an N-dimensional hyper space.

FIG. 3B depicts a method 300B according to one embodiment of step 220.Method 300B operates in substantially the same manner as described abovewith respect to the method 300A of FIG. 3A, except that at each of theevaluation steps 330, 350 and 370, an additional query is provided as towhether the dimensional limit DL has been reached. In each of thesesteps, if the query is answered affirmatively indicating that the DL hasbeen reached (or the CSCMM is a good fit), then the method 300B exits atstep 390.

FIG. 3C depicts a method 300C according to one embodiment of step 220.Method 300C operates in substantially the same manner as described abovewith respect to the method 300A of FIG. 3A, except that at each of theevaluation steps 330, 350 and 370, an additional query is provided as towhether a Constraint violation has occurred or a Constraint would beviolated if the process moves forward by incrementing N in the NFAmodel. In each of the steps, if the query is answered affirmativelyindicating that Constraint adherence has or would be ceased (or that thecurrent SUS-characterizing mathematical model is a good fit), then themethod 300C exits at step 390. A Constraint violation may comprise,illustratively, the reaching of a maximum number of SUS evaluations, thereaching of a maximum amount of time or resources for performing SUSevaluations and so on. In various embodiments, message may be sent toUser or System indicative of a Constraint violation such that the Useror System may responsibly reduce or remove one or more Constraints topermit further processing.

FIG. 3D depicts a method 300D according to one embodiment of step 220.Method 300D operates in substantially the same manner as described abovewith respect to the method 300A of FIG. 3A, except that at one or bothof the 2FA evaluation steps 340-350 and NFA evaluation steps 360-370,additional stress points may be selected or otherwise adapted (e.g., viathe TVE) prior to the evaluation of fit of the current SUS model. Forexample, a different orthogonal array or a different fractionalfactorial design may be selected for each NFA model such that acorrespondingly different set of stress points is appropriate toevaluate the NFA model. As previously noted, stress points may beselected to expose the inability of a model to adequately match the SUSoutput. In various embodiments, SP may be selected using an orthogonalarray. In other embodiments, SP may be selected using fractionalfactorial designs, full factorial designs, design of experiments, spacefilling designs, and/or other method.

New or additional stress points SP may be generated for certain analysissteps and not for others in various embodiments. For example, new oradditional stress points SP may be generated for assessing goodness offit of the 2FA and 5FA models while for assessing goodness of fit of the3FA and 4FA models previously utilized stress points SP may be used.Determination of when stress points SP may be re-used or additional SPmay be generated may depend on available information regarding goodnessof fit in prior analysis steps, Constraints, and other User or Systemspecified criteria. For example if additional SP are generated as shownin the relevant steps, the goodness of fit evaluation may be performedusing both initial and additional SP.

FIG. 3E depicts method 300E according to one embodiment of step 220.Method 300E operates in substantially the same manner as described abovewith respect to the method 300A of FIG. 3A, except that the process canloop back. Method 300E describes an embodiment where there are multiplepasses through the 1FA, 2FA, . . . , NFA processes. In this embodiment,the CSCMM is improved by iteratively employing 1FA, 2FA, . . . , NFAprocesses to progressively improve accuracy in a loop back approach. TheUser or System may specify the progressive maximum orders and loop backcriteria as Constraints for the polynomials in the 1FA, 2FA, . . . , NFAsteps.

Illustratively, loop back criteria may specify that in a first pass, themaximum polynomial order for the 1FA process may be 2, for the secondand for each subsequent pass the maximum polynomial order increases by2, and loop back occurs when the method 300E is at step 335 for the1^(st), 4^(th), and 7^(th) pass. For example, for the first pass themaximum polynomial order for the 2FA process may be 1, and the maximumorder increases by 1 for each subsequent pass, and that loop back occursonly when the process is at step 355 for the 1^(st), 4^(th), and 7^(th)time. For example, for the first pass the maximum polynomial order for3FA process may be 1, and so forth.

A maximum order of 1 for the 2FA process implies that none of theparameters in the two parameter terms may have an order greater than 1.Maximum polynomial orders for 3FA, 4FA, and so are interpretedsimilarly.

For the above illustration, in the first pass, the 1FA process of step320 will improve the CSCMM by adding single parameter polynomials of upto order 2 and the method 300E will progress to step 330. Then thegoodness of fit and constraints will be checked in step 330. If thegoodness of fit and constraint criteria are met, then the method 300Ewill proceed to exit at step 390, else to step 335. Based on the aboveloop back criteria, the method 300E will then move to step 315 where themaximum polynomial order for 1FA process will be increased to 4. Themethod 300E will move to step 320 where the 1FA process will improve theCSCMM by adding single parameter terms up to 4^(th) order. After that,the method 300E will move to step 330. If the goodness of fit andconstraint criteria are not met, then the method 300E will move to step335. This being the second time the method 300E is at step 335, it willnot loop back, instead it will move to step 340.

In this illustration, for the first pass through the 2FA process, themethod 300E will exit step 340 when the order of the 2-parameter termsis 1 and will progress to step 350. If the goodness of fit andconstraints criteria are met, then the method 300E will proceed to exitat step 390, else it will move to step 355 where the loop back criteriawould be checked. Based on the loop back criteria for the illustrationdescribed above, the method 300E will move to step 315 where the maximumorder for the 1FA process will be increased to 6 and maximum order forthe 2FA process will be increased to 2.

In the above illustration, in step 320 single parameter terms of orderup to 6 will be added to the CSCMM and the method 300E will move to step330. In the event that the goodness of fit and constraints criteria arenot met, the method 300E will move to step 335. Being the third pass,the method 300E will move to step 340 where 2-parameter terms of orderup to 2 will be added to the CSCMM. If the goodness of fit criteria andconstraints are not met, the method 300E will move to step 355, and thenmethod 300E will proceed to step 360 as defined by the loop backcriteria.

Similar to step 355, step 375 assesses loop back criteria for 3FA, 4FA,. . . , NFA processes. Per the loop back criteria, when provided, themethod 300E loops back to step 315 and then to step 320.

It is envisioned that in some embodiments the loop back criteria mayspecify to skip/bypass certain steps. Illustratively, if max polynomialorder has been met for 1FA process then step 320 will be skipped.

It will be appreciated by those skilled in the art that the method 300Aof FIG. 3A may be modified in accordance with one or more of themodifications discussed above with respect to the methods 300B-300E of,respectively, FIGS. 3B-3E. That is, it is contemplated that variousembodiments may comprise of combinations or aspects of method 300A,method 300B, method 300C, method 300D, and/or method 300E.

In various embodiments, it is contemplated that other classes offunctions such as Fourier series, wavelets, and Walsh functions may beutilized instead of polynomials. In these embodiments, the variousmethodologies, processes, mechanism and the like operate in asubstantially similar manner as described herein, except for variousmodifications that one skilled in the art would readily appreciate.

FIG. 4A depicts a flow diagram of method 400A of improving the fit ofCSCMM to a SUS according to an embodiment and suitable for use inimplementing step 320 of the methods 300A-300E depicted above withrespect to FIGS. 3A-3E. This is also referred to as a first fitapproximation process or 1FA process.

At step 410 (e.g., via the EQE), the list of candidate input parametersis initialized to all input parameters for the SUS. As previously noted,a dimensional limit DL is defined as the number of input parameters inthe SUS. Thus, at step 410, each of the N input parameters within the DLis identified and selected for further processing as described herein.

At step 410, the 1FA model is initialized to a constant, typically zero,plus an initial approximation model. In various embodiments the initialapproximation model is the latest CSCMM. In various embodiments, theinitial approximation model is as provided by the User or System. Invarious embodiments the initial approximation model is zero. The 1FAmodel is then improved through various steps of method 400A of FIG. 4Adescribed herein to obtain improved approximation for the SUS.

At step 420, ranking stress points RSP are selected (e.g., via the TVE)for the purpose of rank ordering the input parameters. The RSP may begenerated using the same methods utilized for generating SP. In variousembodiments, RSP may be generated using an orthogonal array. In otherembodiments, RSP may be selected using fractional factorial designs,full factorial designs, space filling designs, and/or other methods. Invarious embodiments, RSP selection may be dependent on User input,System input and/or specified Constraints. At step 420, the method 400A(e.g., via the HSSHI) evaluates the SUS at the RSP.

At step 430, the current 1FA model is evaluated at RSP (e.g., via theEQE) and a difference between the evaluations of the SUS and the current1FA model is computed at the corresponding RSP.

At step 440, the main effects (single parameter effects) of all inputparameters are computed (e.g., via the EQE). That is, a computation ismade to determine the main effects of all input parameters on thedifference between the SUS and current 1FA model (i.e., SUS minuscurrent 1FA model). In various embodiments, sum of squares, F-statisticsand/or other statistics are also computed. Main effects, sum of squares,F-statistics and/or other statistics may be used to define theimportance and/or rank order of the input parameters. One skilled in theart will appreciate that computing main effects, sum of squares,F-statistics and the like are standard statistical methods in design ofexperiments and regression analysis. The input parameters may be rankordered by importance. Alternatively, the input parameters can also berank ordered per other criteria, such as presumed importance (ratherthan actual importance) based upon prior importance data or otherinformation, User or System inputs, order of input parameter entry,random order, alphabetical order or other criteria.

At step 450, the method 400A (e.g., via the EQE) selects the inputparameter from among the candidate parameters having the highest rank(illustratively, with the largest effect) as the current dominant inputparameter. This selection may be based on the sum of squares,F-statistic values for the input parameters, and/or other criteria.

At step 450, the method 400A also examines if the effect of an inputparameter that was previously selected as a dominant input parameter islarge compared to the effects of the current candidate parameters. Ifso, the aliasing property of the RSP may be used to identify potentialinteracting input parameters in step 750 of method 700A, describedbelow, for selecting a dominant input parameter pair. Alternatively, ifthe effect of an input parameter that was previously selected as adominant input parameter is large compared to the effects of the currentcandidate parameters, that information may be used in determining thepolynomial order for that input parameter in future iterations.

At step 460, method 400A (e.g., via the EQE) generates a polynomial forthe current dominant input parameter for use in the 1FA. It is notedthat the order of polynomial, associated test vectors, polynomialestimation points and the like associated with the current dominantinput parameter may be selected as described herein with respect to thevarious figures. In various embodiments, the polynomial may beconstructed with origin at or near the center point CP. In variousembodiments, where origin is not at the center point CP, the polynomialmay be transformed so the origin is at or near the center point CP.

At step 470, the method 400A (e.g., via the EQE) updates the current 1FAmodel by adding the polynomial for the current dominant input parametergenerated in step 460. Here the constant term is adjusted, if necessary,so that the evaluations of the SUS and the updated 1FA model match atCP. In various embodiments, the constant term may be adjusted so thatthe evaluations of SUS and the updated 1FA model match at a suitablepoint in the region of interest RI. Also, the current dominant inputparameter is removed from the candidate input parameter list.

At step 480, the method 400A (e.g., via the EQE) evaluates the current1FA model at the previously selected stress points from step 310 todetermine whether the 1FA model represents a good fit to the SUS, suchas described herein with respect to the various figures. If the 1FAmodel represents a good fit, then the method 400A exits at step 490.Otherwise, the method 400A proceeds to step 485.

At step 485, the method 400A (e.g., via the EQE) makes a query as towhether all DL input parameters have been processed (i.e., there are nomore input parameters in the candidate input parameter list). If yes,the method 400A exits at step 490. If not, the process loops back tostep 430. The loop continues until a good fit is attained or all inputparameters are exhausted. In each loop, steps 430-480 are repeated.

It is noted that less than DL input parameters may be used to generatethe final 1FA model if the subset of DL input parameters processedprovide a 1FA model of sufficient accuracy.

At step 490, the method 400A sets CSCMM equal to the latest 1FA modelfrom the latest pass through the step 470, which is an improvedapproximation model. Method 400A exits at step 490.

FIG. 4B depicts a flow diagram of method 400B of improving the fit ofCSCMM to a SUS according to an embodiment and suitable for use inimplementing step 320 of the methods 300A-300E depicted above withrespect to FIGS. 3A-3E. This is also referred to as a first fitapproximation process or 1FA process. The steps of FIG. 4B are similarto FIG. 4A with the exception of step 485. If all input parameters arenot processed, the method 400B moves to step 450. In this embodiment,prioritization of input parameters is performed one time.

In various embodiments of method 400B of FIG. 4B, steps 420 and 430 areskipped. In step 440 calculations of main effects are also skipped andrank ordering is done in random order, in the order the input parameterswere entered by the user, or a convenient order. The step 480 isskipped, so that after step 470, the method 400B moves directly to step485. In these embodiments, the method 400B builds a polynomial model forall input parameters and includes them in the 1FA model.

In various embodiments of methods 400A and 400B of FIGS. 4A and 4B,respectively, the goodness of fit criteria of step 480 may be adaptedsuch that goodness of fit is assessed only when a specified set of inputparameters is included in the 1FA model. That is, in the cases thatpolynomials for a specified set of input parameters have not beenincluded in the 1FA model, in various embodiments the step 480 isadapted such that the check for goodness of fit is set to “not a goodfit” and the method proceeds to step 485.

A specified set of input parameters may comprise all input parameters, asubset of the input parameters, or one input parameter and may bedetermined by User input, System input, default settings, orConstraints. In these various embodiments, step 420 and/or step 430 areadapted such that they may be skipped. In various embodiments, step 440may be adapted such that main effects of input parameters are notcomputed and rank ordering is determined by presumed importance, pastdata, random order, user or system specification, alphabetical order, oranother convenient method as specified by User, System or default.

FIG. 5 depicts a flow diagram of a method of generating a fittedpolynomial model for an input parameter associated with a SUS. Themethod 500 of FIG. 5 is suitable for use in the various other methodsdiscussed herein with respect to the various figures.

At step 510, a plurality of points, illustratively n points, areselected (e.g., via the TVE) along the Axis Parallel Line APL for thecurrent dominant input parameter. These points are referred to as APLpoints and also as modeling generation points since they are used forfitting a polynomial model. In essence, these model generation pointsare 1-dimensional grid points. The current dominant input parameter isalso referred to as the selected input parameter. An Axis Parallel LineAPL for an input parameter is denoted herein as a line parallel to theaxis of that input parameter which passes through the Center Point CP.

Here, n is defined as a positive integer larger than 1 (e.g., 2, 3, . .. ). n may be the same or different for each input parameter. APL pointsmay be positioned to span a region of interest RI based on probabilitydistribution associated with the input parameter or its specified regionof interest. Further, a starting (e.g., initial) value of n or a maxvalue of n may be derived from User input, System input, or otherinformation.

In various embodiments, the n points on the APL may be equally spacedfrom each other and symmetrically distributed with respect to the CP sothat the centroid of these points coincides with the CP. In variousembodiments, the n points may be positioned along the APL of an inputparameter in a manner intended to span the region of interest for theparameter to a significant degree (e.g., 20%, 60%, 80%, or other percentof the RI). For example, FIG. 6A graphically depicts a n-point (heren=3) distribution where three equispaced points at locations x₁, x₂, andx₃ may be selected to be symmetrically positioned with respect to theCenter Point CP. Let x□ denote the average value of the selected points.Here x□ is equal to (x₁+x₂+x₃)/3. Thus, in this example, coincides withthe x coordinate of CP. In another example depicted in FIG. 6B, fourpoints are shown selected for the same parameter. These points arepositioned at x₁, x₂, x₃, and x₄.

At step 520, the method 500 evaluates the SUS (e.g., via the HSSHI) atthe selected APL points in step 510.

At step 530, the method 500 fits or generates a (n−1) order polynomialby solving n equations in n unknowns for the current dominant inputparameter. Various processes and manipulations illustrative ofembodiments of step 530 will now be described as being performed usingthe Equation Engine 121-EQE discussed above with respect to FIG. 1,though other functional elements may also perform these steps.

Generating a Polynomial Model for an Input Parameter

For example, and referring to the three points depicted in FIG. 6A forinput parameter x, the method 500 generates three vectors (e.g., via theTVE) corresponding to the three points. Each vector has DL componentscorresponding to the values of the DL input parameters. The points maybe positioned along the APL for parameter x at the coordinate values x₁,x₂, and x₃; and the spacing between adjacent points is h. The average ofx₁, x₂, and x₃ is x□, which is equal to the x coordinate of CP. Thevalues of the other input parameters may be set at the respectivecoordinates of the CP. The method 500 then evaluates the SUS (e.g., viathe HSSHI) at the three points at x₁, x₂, and x₃ to provide therebyevaluation results as y₁, y₂, and y₃ respectively. An exemplarypolynomial to be fitted in this example is provided by the followingequation:)

y=b ₀ +b ₁(x−x□)+b ₂(x−x□)²  (Eq. 1)

Polynomials of other types/structures and other classes of functions mayalso be used for this purpose.

The three SUS evaluations provide the following three simultaneousequations (Eqs. 2A-2C):

y ₁ =b ₀ b ₁(x ₁ −x□)b ₂(x ₁ <x□)²  (Eq. 2A)

y ₂ =b ₀ +b ₁(x ₂ −x□)+b ₂(x ₂ −x□)²  (Eq. 2B)

y ₃ =b ₀ +b ₁(x ₃ −x□)+b ₂(x ₃ −x□)²  (Eq. 2C)

These three linear equations in three unknowns (b₀, b₁, and b₂) may besolved by any one of the standard methods of solving simultaneous linearequations, such as matrix inversion, successive substitution, Lagrangepolynomials and the like.

In an alternate embodiment, a polynomial is fit/generated using apolynomial expressed as Eq.1 rewritten as a sum of orthogonalpolynomials. For example, one approach is to construct the Vandermondematrix, which for the embodiment with three points is shown below in Eq.3a:

$\begin{matrix}{V = \begin{bmatrix}1 & \left( {x_{1} - {x\; \bullet}} \right) & \left( {x_{1} - {x\; \bullet}} \right)^{2} \\1 & \left( {x_{2} - {x\; \bullet}} \right) & \left( {x_{2} - {x\; \bullet}} \right)^{2} \\1 & \left( {x_{3} - {x\; \bullet}} \right) & \left( {x_{3} - {x\; \bullet}} \right)^{2}\end{bmatrix}} & \left( {{{Eq}.\mspace{14mu} 3}a} \right)\end{matrix}$

Then, the columns of the matrix V are successively orthogonalized (e.g.,via the EQE) leading to the following matrix V* as shown below in Eq.3b:

$\begin{matrix}{V^{*} = \begin{bmatrix}1 & \left( {x_{1} - {x\; \bullet}} \right) & {\left( {x_{1} - {x\; \bullet}} \right)^{2} - {\left( {2/3} \right)h^{2}}} \\1 & \left( {x_{2} - {x\; \bullet}} \right) & {\left( {x_{2} - {x\; \bullet}} \right)^{2} - {\left( {2/3} \right)h^{2}}} \\1 & \left( {x_{3} - {x\; \bullet}} \right) & {\left( {x_{3} - {x\; \bullet}} \right)^{2} - {\left( {2/3} \right)h^{2}}}\end{bmatrix}} & \left( {{{Eq}.\mspace{14mu} 3}b} \right)\end{matrix}$

The column vectors of the V* matrix are mutually orthogonal and implythe following orthogonal polynomials:

P ₀(x−x□)=I  (Eq. 4)

P ₁(x−x□)=(x−x□)  (Eq. 5)

P ₂(x−x□)=(x−x□)²−(2/3)h ²  (Eq. 6)

Using the orthogonal polynomials, the method 500 adapts Eq. 1 as a sumof orthogonal polynomials such as shown below in Eq. 7, where B; arecoefficients:

y=Σ _(i=0 to 2) {B _(i) P _(i)(x−x□)}  (Eq. 7)

To calculate the coefficient B_(α), where α is 0, 1, or 2, the method500 multiplies both sides of Eq. 7 by P_(α)(x−x□) and sums both sidesover the sample points x₁, x₂, and x₃. The method 500 then solves theresulting equation for B_(α) for α=0 to 2, to provide the result shownbelow as Eq. 8:

B _(α)=(numerator_(—) Eq8)/(denominator_(—) Eq8)  (Eq. 8)

where

-   -   numerator Eq8=Σ_(j=1 to 3) {y_(j) P_(α)(x_(j)−x□)}; and    -   denominator Eq8=Σ_(j=1 to 3) {P_(α) ²(x_(j)−x□)}

In an embodiment graphically depicted with respect to FIG. 6C, thepoints x₁, x₂, and x₃ are not equally spaced. For that embodiment,method 500 (e.g., via the EQE) uses one of the several methods describedherein for fitting a polynomial. For example, method 500 (e.g., via theEQE) may use Eq. 1 to represent the polynomial and solve Eq. 2A-2C tocompute the coefficients b₀, b₁, and b₂ by using the prescribed methods.

Those skilled in the art will readily note that the terms fit apolynomial, generate a polynomial, and construct a polynomial refer tosubstantially the same process and may be used interchangeably.

In the embodiment with four points (n=4) shown in FIG. 6B, at step 530,the method (e.g., via the EQE) fits an (n−1) order polynomial by solvingn equations in n unknowns. The method 500 (e.g., via the TVE) generatesfour vectors corresponding to the four points. Each vector has DLcomponents corresponding to the DL input parameters. The points may bepositioned along the APL for the parameter x at the coordinate valuesx₁, x₂, x₃, and x₄ with a spacing of h between adjacent points. Theaverage of x₁, x₂, x₃, and x₄ is x□, which is equal to the x coordinateof CP. The values of the other input parameters may be set at therespective coordinates of the CP. The method 500 (e.g., via the HSSHI)evaluates the SUS at the four points and the evaluation results aredenoted as y₁, y₂, y₃, and y₄ respectively. The polynomial to be fittedfor this embodiment may be expressed by the following equation:

y=b ₀ +b ₁(x−x□)+b ₂(x−x□)² +b ₃(x−x□)³  (Eq. 9)

The four SUS evaluations provide the following four simultaneousequations:

y ₁ =b ₀ b ₁(x ₁ −x□)+b ₂(x ₁ −x□)² +b ₃(x ₁ −x□)³  (Eq. 10A)

y ₂ =b ₀ +b ₁(x ₂ −x□)+b ₂(x ₂ −x□)² b ₃(x ₂ −x□)³  (Eq. 10B)

y ₃ =b ₀ b ₁(x ₃ −x□)+b ₂(x ₃ −x□)² b ₃(x ₃ −x□)³  (Eq. 10C)

y ₄ =b ₀ +b ₁(x ₄ −x□)+b ₂(x ₄ −x□)² +b ₃(x ₄ −x□)³  (Eq. 10D)

The method 500 (e.g., via the EQE) solves these four linear equations infour unknowns (b₀, b₁, b₂, and b₃) by one of the standard methods ofsolving simultaneous linear equations. These methods include matrixinversion, successive substitution, Lagrange polynomials, and other.

Another method utilized by the method 500 to fit a polynomial is torewrite Eq. 9 as an orthogonal polynomial expansion. One approach is toconstruct the Vandermonde matrix (denoted as matrix V), and thensuccessively orthogonalize the columns of the matrix V to generatematrix V. The column vectors of matrix V* are mutually orthogonal andimply the following orthogonal polynomials (Eqs. 11A-11D). The detailsof this process are substantially the same as the above-describedembodiment with three points. In particular, the following equations fororthogonal polynomials are provided for this 4 point example:

P ₀(x−x□)=1  (Eq. 11A)

P ₁(x−x□)=(x−x□)  (Eq. 11B)

P ₂(x−x□)=(x−x□)²−(5/4)h ²  (Eq. 11C)

P ₃(x−x□)=(x−x□)³−(41/20)(x−x□)h ²  (Eq. 11D)

Using the orthogonal polynomials, the method 500 expresses Eq. 9 in theform of an orthogonal polynomial expansion, as shown below with respectto Eq. 12, where and B; for i=0, 1, 2, and 3 are the coefficients.

y=Σ _(i=0 to 3) {B _(i) P _(i)(x−x□)}  (Eq. 12)

To calculate the coefficients B_(α), where α is 0, 1, 2, or 3, themethod 500 multiplies both sides of Eq. 12 by P_(α)(x−x□) and sums bothsides over the points x₁, x₂, x₃, and x₄. Then the method 500 solves theresulting equation for B_(α) for α=0 to 3.

In various embodiments, illustratively while conducting hardwareexperiments/tests, evaluations of the SUS may be conducted at pointsnear the specified modeling points rather than exactly at the specifiedmodeling points.

In various embodiments, illustratively for convenience of evaluation ofthe SUS or other reasons, the model generation points may be selected asindicated above with some offset causing the model generation points tobe close to the axis parallel line, but not exactly on the axis parallelline. In various embodiments, for subsequent analyses, such asgeneration of polynomials, the model generation points may be treated asthough they are at their respective closest locations on the APL. Invarious embodiments, for subsequent analyses the actual coordinates ofthe model generation points may be used in conjunction with generallinear models analysis.

At step 540 the method 500 (e.g., via the TVE) generates one or moreadditional points and corresponding vectors with DL componentscorresponding to the values of the DL input parameters. The additionalpoints, called Axis Parallel Line Stress Points ASP, may be positionedalong the APL for the input parameter x within the RI and may bedistinct from the points used for generating the polynomial model in thepreceding step 530. For example, in the embodiment with 3 points shownin FIG. 6A, the additional point(s) would be distinct from the pointsA₁, A₂, and A₃ (positioned at x₁, x₂, and x₃ respectively). The valuesof the other input parameters may be set at the respective coordinatesof the CP.

At step 550, the method 500 (e.g., via the HSSHI) evaluates the SUS andthe polynomial model for x generated in step 530 at the ASP, and then(e.g., via the EQE) evaluates the difference between the evaluations ofthe SUS and the polynomial model for x generated in step 530 at thecorresponding ASP.

At step 560, the adequacy of the polynomial model for input parameter xgenerated in step 530 is tested by method 500. That is, at step 560 adetermination is made as to whether the generated model for the currentdominant input parameter is a good fit with respect to the evaluationsof the SUS along the APL for that parameter. If the difference computedin step 550 is within a User, System or default specification, then thepolynomial order is considered adequate (i.e., a good fit) and themethod 500 proceeds to step 590. If the difference is outside of theUser, System or default specification (i.e., not a good fit), then themethod 500 proceeds to step 570.

At step 570, a determination is made by method 500 (e.g., via the EQE)as to whether a maximum polynomial order has been reached. If themaximum polynomial order has not been reached, then the method proceedsto step 580. Otherwise the method 500 optionally provides a message tothe User (e.g., via the HSSHI) and proceeds to step 590. In variousembodiments, the User or the System may specify the polynomial order touse, in which case, both the starting polynomial order and the maximumpolynomial order are set equal to the specified polynomial order. Forexample, if the polynomial order was specified as 4, n would be set to 5in order to fit a (n−1) order polynomial, i.e., a 4^(th) orderpolynomial. This may be specified in the default settings,configurations, or by means of Constraints.

At step 580, method 500 (e.g., via the EQE) increases the polynomialorder by increasing the number of points along the APL of the currentdominant input parameter by one or more such that the correspondingpolynomial order is increased to the new number of points minus one.Steps 510-560 are then repeated until the difference between the SUS andpolynomial model evaluations is within User, System or defaultspecifications.

At step 590, method 500 (e.g., via the EQE) removes the constant termfrom the polynomial model generated in latest step 530.

At step 590, in the case that the origin of the coordinate system is notat the CP, the polynomial built in the latest step 530 is transformed byshifting the origin to the center point CP; constant term is removedfrom the transformed polynomial; and the resulting polynomial istransformed again by shifting the origin from CP back to the originalorigin.

Optionally at step 590, the method 500 simplifies the polynomial byevaluating the contribution of each remaining polynomial term andremoving any term that does not provide sufficient contribution asspecified by the User or System.

The method 500 exits at step 590.

In various embodiments, step 570 is adapted such that the check forattaining maximum polynomial order is set to “yes” so that method 500moves from step 570 to step 590 and does not go to step 580. That is,method 500 is adapted to not iterate and increase the polynomial order.

In various embodiments, step 540 is adapted such that ASP are notselected. That is, method 500 is adapted to skip step 540. In variousembodiments, step 550 is adapted such that computation of differencebetween the SUS and the polynomial at ASP is not done. That is, method500 is adapted to skip step 550. In various embodiments, step 560 isadapted such that the check for good fit is set to “yes” so that method500 moves from step 560 to step 590. That is, method 500 is adapted tonot iterate and increase the polynomial order.

For example, in an embodiment with two input parameters u and v, bothbeing approximated by second order polynomials, the summed polynomial atstep 470 of methods 400A-400B of, respectively, FIGS. 4A-4B may take thefollowing form:

y=b ₀₀ +b ₁₁(u−u□)+b ₁₂(u−u□)² +b ₂₁(v−v□)+b ₂₂(v−v□)²  (Eq. 13)

Here u□ is the average of the u coordinates of the three points selectedin computing the polynomial for input parameter u in step 460. Likewise,v□ is the average of the v coordinates of the three points selected incomputing the polynomial for input parameter v in step 460. Here b₁₁,b₁₂, b₂₁, and b₂₂ are coefficients computed in step 460.

The constant b₀₀ is such that the polynomial (Eq. 13) evaluation matchesthe SUS evaluation at the CP.

It should be noted that the various input parameters could use the sameorder polynomials or different order polynomials. That is, it is notnecessary for the all input parameters to have the same orderpolynomial. For example, u could have a 3^(rd) order polynomial (i.e., aterm with u³) and v could have a 4^(th) order polynomial (i.e., a termwith v⁴).

FIG. 7A depicts a flow diagram of method 700A of improving the fit ofCSCMM to a SUS according to an embodiment and suitable for use inimplementing step 340 of the methods 300A-300E depicted above withrespect to FIGS. 3A-3E. This is also referred to as a second fitapproximation process or 2FA process. In various embodiments, the method700A of FIG. 7A is adapted to provide an improved CSCMM in which theinput parameter pairs having the most effect on the model may besuccessively selected for use in generating polynomials used within theimproved CSCMM.

At step 710, the method 700A (e.g., via the EQE) initializes thecandidate input parameter pair list to all possible input parameterpairs.

At step 710, the 2FA model is initialized to a constant, typically zero,plus an initial approximation model. In various embodiments the initialapproximation model is the latest CSCMM. In various embodiments, theinitial approximation model is as provided by the User or System. Invarious embodiments the initial approximation model is zero. The 2FAmodel is then improved through various steps of method 700A of FIG. 7Adescribed herein to obtain improved approximation model for the SUS.

At step 720, the method 700A (e.g., via the TVE) generates a pluralityof points within the region of interest RI of the input parameter spacefor the purpose of rank ordering the input parameter pairs. These pointsare referred as ranking stress points RSP. The RSP of method 700A may ormay not be the same as those for the methods 400A-400B. The RSP aregenerated using the same methods utilized for generating SP, namely,they could be generated using orthogonal arrays, fractional factorialdesigns, full factorial designs, or other methods. However, the RSP neednot be the same as the SP. For example, SP may be generated using2-level orthogonal arrays and RSP may be generated using 3-levelorthogonal arrays. At step 720, the method 700A (e.g., via the HSSHI)evaluates the SUS at the RSP.

At step 730, the method 700A (e.g., via the EQE) evaluates the current2FA model at the RSP and then computes the difference between the SUSevaluations and the current 2FA model evaluations at the correspondingRSP (SUS evaluation minus the 2FA evaluation).

At step 740, the method 700A (e.g., via the EQE) computes the 2parameter interactions for each input parameter pair using thedifference in the evaluations computed in step 730. The interaction foreach pair of parameters may be summarized as the largest absolutedifference, sum of absolute differences, mean square of selected topabsolute differences, mean square error, or other.

In various embodiments, interaction values may be calculated for allpairs of input parameters. In one embodiment where there are 4 inputparameters (labeled u, v, w, and x) the process may select 8 RSPaccording to an orthogonal array L8 of Table 1 leading to the 8 vectorsaccording to Table 2. Assume y₁* for i=1 to 8 are the computeddifferences in step 730 corresponding to the 8 RSP. The process computesthe level pair average for parameters u and v at (u₁, v₁) by identifyingthe stress points where the level pair (u₁, v₁) occurs, and averagingthe corresponding y₁* values. Here, the level pair average for (u₁, v₁)is given by L(u₁, v₁) as follows:

L(u ₁ ,v _(i))=(y ₁ *+y ₂*)/2  (Eq. 14)

The other level pair averages of u and v at (u₁, v₂), (u₂, v₁), and (u₂,v₂) are computed similarly. Likewise level pair averages for all pairsof input parameters are calculated. In this embodiment there are 6possible pairs of input parameters and for each pair there are 4possible level pairs. Accordingly, the number of computations will be 4times 6, which is equal to 24 in this example.

In various embodiments, the level pair averages define the importance ofinput parameter pairs.

At step 750 (e.g., via the EQE), the input parameter pairs are rankordered by importance. Alternatively, the input parameters can also berank ordered per other criteria, such as presumed importance (ratherthan actual importance) based upon prior importance data or otherinformation, User or System inputs, order of input parameter entry,random order, alphabetical order or other criteria. The input parameterpair with the largest rank (e.g., largest interaction effect), denotedillustratively herein as u and v, is selected from the current candidateinput parameter pairs.

In various embodiments, the largest absolute value among the 24 valuesis selected to decide the current dominant interaction pair. Interactionpair is also referred as input parameter pair. Illustratively, ifabsolute value of L(u_(i), w₁) is the largest, then the input parameterpair (u, w) would be selected as the current dominant interaction pair.At step 750 (e.g., via the EQE), for this embodiment, if the absolutevalue of L(v₁, w₂) is the largest, then the input parameter pair (v, w)would be selected as the current dominant interaction pair.

In various embodiments, the process calculates the mean sum of square ofthe 4 level pair averages associates with each input parameter pair.Thus, for input parameter pair (u, v) the process calculates the meansum of squares, S(u, v), as follows:

S(u,v)={L(u ₁ ,v ₁)² +L(u ₁ ,v ₂)² +L(u ₂ ,v ₁)² +L(u ₂ ,v ₂)²}/4  (Eq.15)

There would be 6 such mean sum of square values corresponding to the 6input parameter pairs. In various embodiments mean sum of square valuesof level pair averages for specific parameter pairs define importance.The largest importance determines the current dominant interaction pair.

The current dominant interaction pair is also referred to as currentdominant input parameter pair, or selected input parameter pair, orselected interaction pair or the like.

At step 760, the method 700 (e.g., via the EQE) constructs a twoparameter polynomial involving the current dominant input parameterpair, illustratively depicted by u and v. It is noted that the orders of2-parameter polynomial terms, associated test vectors, polynomialestimation points and the like associated with the current dominantinput parameter pair may be selected as described herein with respect tothe various figures. In various embodiments, the polynomial may beconstructed with origin at or near the center point CP. In variousembodiments, where origin is not at the center point CP, the polynomialmay be transformed so the origin is at or near the center point CP.

At step 770, the method 700A updates the current 2FA model by adding thepolynomial for the current dominant input parameter pair generated instep 760. Here the constant term is adjusted, if necessary, so that theevaluations of the SUS and the updated 2FA model match at CP. In variousembodiments, the constant term is adjusted so that the evaluations ofSUS and the updated 2FA model match at a suitable point in the region ofinterest RI. Also, the current dominant input parameter pair is removedfrom the candidate input parameter pair list.

At step 780, the method 700A (e.g., via the EQE) evaluates the current2FA model (updated in step 770) at stress points SP. The evaluations ofthe 2FA model at the SP are compared to the evaluations of the SUS atthe corresponding SP. The method 700 measures “goodness” of fit by usinga maximum absolute difference, a sum of absolute differences, a meansquare of selected top absolute differences, a mean square error, orother techniques. If fit is within User specified, System specified ordefault limits, the method 700A exits at step 790. If the fit is outsideUser specified, System specified, or default limits, then the method700A proceeds to step 785.

At step 785, the method 700A (e.g., via the EQE) determines whether allcandidate input parameter pairs have been exhausted (i.e., the candidateinput parameter list is empty). If all input parameter pairs areexhausted, the method 700A proceeds to step 790. If all input parameterpairs are not exhausted, then the method loops back to step 730. Steps730-780 are executed until good fit is attained or all candidate inputparameter pairs are exhausted, at which point the method exits at step790.

At step 790, the method 700A sets CSCMM equal to the latest 2FA modelfrom the latest pass through the step 770, which is an improvedapproximation model. Method 700A exits at step 790.

FIG. 7B depicts a flow diagram of method 700B of fitting a 2FA model,second fit approximation, to a SUS (2FA process) according to anembodiment and suitable for use in implementing step 340 of the methods300A-300E depicted above with respect to FIGS. 3A-3E. The steps 710 to780 are the same as described in method 700A. In the case that all pairsof parameters are not exhausted at step 785, the method 700B proceeds tostep 750, instead of step 730 as depicted in method 700A. In method700B, interaction and prioritization analysis is computed one time andthe results are used for all iterations of pairs of input parameters asopposed to in method 700A where interaction and prioritization analysisis computed each time a new input parameter pair is to be processed andremoved from the candidate input parameter list.

In various embodiments of methods 700A and 700B of FIGS. 7A and 7B,respectively, the goodness of fit criteria of step 780 may be adaptedsuch that goodness of fit is assessed only when a specified set of inputparameter pairs is included in the 2FA model. That is, in cases wherepolynomials for a specified set of input parameter pairs have not beenincluded in, illustratively, the 2FA model, in various embodiments thestep 780 is adapted such that the check for goodness of fit is set to“not a good fit” and the method proceeds to step 785. A specified set ofinput parameter pairs may comprise all input parameter pairs, a subsetof the input parameter pairs, or one input parameter pair and may bedetermined by User input, System input, default settings, orConstraints. In these various embodiments, step 720 and/or step 730 maybe skipped. In various embodiments, step 740 may be adapted such thateffects of input parameter pairs are not computed and rank ordering isdetermined by presumed importance, past data, random order, user orsystem specification, alphabetical order, or another convenient methodas specified by User, System, or default.

In various embodiments, methods 700A-700B are adapted so step 780 isskipped. The method 700A or 700B moves from step 770 directly to step785. That is, in various embodiments, all input parameter pairs areprocessed.

It will be appreciated by one skilled in the art that methods 700A-700Bare similar in process and computation to methods 400A-400B except thatthe polynomials terms generated in methods 400A-400B involve oneparameter and polynomials terms generated in methods 700A-700B involvetwo parameters.

FIG. 8 depicts a flow diagram of a method of generating a 2-parameterpolynomial for a selected input parameter pair according to oneembodiment. Specifically, the method 800 of FIG. 8 (and variousmodifications thereto) is suitable for use in implementing step 760 ofthe methods 700A-700B of FIGS. 7A-7B.

At step 810, the method 800 (e.g., via the TVE) sets up, illustratively,a K1 by K2 grid centered on the CP in the plane of the axis parallellines APLs for the input parameters u and v identified in step 750.Here, K1 and K2 are positive integers greater than or equal to 2. Thegrid has K1 points along the APL for parameter u and K2 points along theAPL for parameter v.

FIGS. 9A-9B graphically depict two-dimensional n-point distributionsuseful in understanding the various embodiments.

FIG. 9A graphically depicts an embodiment where K1=2 and K2=2. The gridpoints are labeled for i=1, 2 and j=1, 2. Next, method 800 (e.g., viathe TVE) generates vectors with DL components corresponding to thevalues of the DL input parameters for each of the K1 by K2 (K1×K2) gridpoints. The values of the input parameters u and v may be set equal tothe corresponding coordinates of the grid points. The values of theother input parameters may be set at the corresponding coordinates ofthe CP.

The grid points are also referred to as model generation points as theymay be used for fitting a polynomial model. In essence, these modelgeneration points may be 2-dimensional grid points.

At step 820, the method 800 (e.g., via the HSSHI) evaluates the SUS atthe grid points. For example, let y_(ij) denote the SUS values at A_(ij)for i=1, 2 and j=1, 2.

At step 830, the method 800 (e.g., via the EQE) computes the twoparameter polynomial. For the above example, the polynomial to be fittedis expressed as follows:

y=Σ _(q=0 to 1)Σ_(r=0 to 1) {b _(qr)(u−u□)^(q)(v−v□)^(r)}  (Eq. 16)

where b_(qr) for q=0, 1 and r=0, 1 are coefficients; u□ is the averageof u₁, and u₂; and v□ is the average of v₁, and v₂.

The four SUS evaluations y_(ij) provide four linear equations,represented by Eq.17, in four unknowns, as follows:

y _(ij)=Σ_(q=0 to 1)Σ_(r=0 to 1) {b _(qr)(u _(i) −u□)^(q)(v _(j)−v□)^(r)}  (Eq. 17)

-   -   for i=1, 2 and j=1, 2.

It will be appreciated by one skilled in the art that Eq. 17 representsfour equations corresponding to (i=1 and j=1), (i=1 and j=2), (i=2 andj=1), and (i=2 and j=2).

The method 800 (e.g., via the EQE) solves these equations by thestandard methods of solving simultaneous linear equations such as matrixinversion, successive substitution, and others.

Another technique that may be utilized to fit a polynomial is to rewriteEq. 17 as a sum of the product of orthogonal polynomials, such as perthe following equation, where P_(q) and Q_(r) are orthogonalpolynomials, such as determined by the EQE using the method describedabove with respect to the 1FA process of method 500 of FIG. 5:

y=Σ _(q=0 to 1)Σ_(r=0 to 1) {B _(qr) P _(q)(u−u□)Q _(r)(v−v□)}  (Eq. 18)

where

P ₀(u−u□)=1  (Eq. 19A)

P ₁(u−u□)=(u−u□)  (Eq. 19B)

Q ₀(v−v□)=1  (Eq. 19C)

Q ₁(v−v□)=(v−v□)  (Eq. 19D)

To calculate the coefficients B_(αβ) (for α=0, 1 and β=0, 1), EqEmultiplies both sides of Eq. 18 by P_(α)(u−u□) and Q_(β)(v−v□) and sumsboth sides over all grid points. The solution for B_(αβ) is:

B _(αβ)=(numerator_(—) Eq20)/(denominator_(—) Eq20)  (Eq. 20)

-   -   where    -   numerator_Eq20=Σ_(i=1 to 2) Σ_(j=1 to 2){y_(ij) P_(α)(u_(i)−u□)        Q_(β)(v_(j)−V□)}    -   denominator_Eq20={Σ_(i=1 to 2) P_(α) ²(u_(i)−u□)}{Σ_(j=1 to 2)        Q_(β) ²(v_(j)−v□)}

In various embodiments, illustratively for convenience of evaluation ofthe SUS, the model generation points may be selected as indicated abovewith some offset causing the model generation points to be close to theplane of the axis parallel lines, but not exactly on the plane. Insubsequent analyses, such as generation of polynomials, the modelgeneration points may be treated as though they are exactly on the planeof the axis parallel lines at the specified locations. In variousembodiments, for subsequent analyses the actual coordinates of the modelgeneration points are used in conjunction with general linear modelsanalysis.

At step 840, the method 800 (e.g. via the TVE) selects one or more OffGrid Stress Points OGSP, which are points in the plane of the APLs ofinput parameters u and v that do not coincide with the grid pointsgenerated in step 810, that are suitable to assess goodness of the2-parameter polynomial generated in step 830 for mathematicallycharacterizing the SUS in the plane of the APLs of the input parametersu and v.

At step 850 the method 800 (e.g., via the HSSHI) evaluates the SUS andthe 2-parameter polynomial at the OGSP.

In various embodiments such as where the SUS is a hardware system, theevaluations of the SUS at the model generation points, SP, ASP, OGSP,APL points, grid points, RSP, and the like may be repeated (e.g. SUSevaluated more than once at the relevant points). In these embodiments,the average of the evaluations at each point and/or the individual datavalues may be used for model building or evaluations of fit, asappropriate.

At step 850, the method 800 (e.g., via the EQE) computes the differencebetween evaluations of the SUS and evaluations of the 2-parameterpolynomial computed in step 830 at the corresponding OGSP.

At step 860, the method 800 (e.g., via the EQE) evaluates if thedifference computed in step 850 is indicative of a good fit (i.e.,within User, System or default specifications). If it is a good fit,then the method 800 proceeds to exit at step 890. Otherwise, the method800 proceeds to step 870.

At step 870, the method 800 determines if a maximum grid size has beenreached. That is, in some embodiments, a User specified, Systemspecified or default maximum polynomial order is provided via,illustratively, default settings, configurations, or by means ofConstraints. If a maximum polynomial order is specified, the processchecks the limits in step 870. If maximum order is reached, the method800 moves to step 890. If maximum polynomial order is not reached, themethod 800 moves to step 880. At step 880 the grid size is incrementedby increasing K1, K2, or both by one or more (making sure that K1 and K2stay within their respective limits) and the process loops back to step810.

In various embodiments, the User or the System may specify thepolynomial order to use for u or v or both, in which case, both thestarting polynomial order and the maximum polynomial order for therespective input parameter(s) are set equal to the specified polynomialorder. For example, if the polynomial order was specified as 4 for u and6 for v, the starting and maximum values of K1 would be set to 5, andstarting and maximum values of K2 would be set to 7. It is noted thatthe specified orders of u and v can be equal. This may be specified inthe default settings, configurations, or by means of Constraints.

At step 890, method 800 removes all single parameter terms and theconstant term from the polynomial model generated in the latest step830.

At step 890, in the case that the origin is not at the CP, thepolynomial built in the latest step 830 is transformed by shifting theorigin to the center point CP; constant term and single parameter termsin the new coordinates may be removed from the transformed polynomial(i.e., any term that does not involve both transformed input parametersmay be removed); and the resulting polynomial is transformed again byshifting the origin back from the CP to the original origin.

Optionally at step 890, the method 800 simplifies the remainingpolynomial by evaluating the contribution of each remaining polynomialterm and removing any term that does not provide sufficient contributionas specified by the User or System. In various embodiments, the method800 is adapted such that at step 890, single parameter terms (e.g. termsthat do not involve both the parameters in the selected input parameterpair) are not removed. In various embodiments, step 890 may not removesingle parameter terms for one iteration, but will remove singleparameter terms in another iteration.

At step 890 the method 800 exits.

In various embodiments, step 870 is adapted such that a check flagindicative of attaining maximum grid size is set to “yes” such thatmethod 800 moves from step 870 to step 890 and does not go to step 880.That is, method 800 is adapted to not iterate and increase thepolynomial orders.

In various embodiments, step 840 is adapted such that OGSP are notselected. That is, method 800 is adapted to skip step 840. In variousembodiments, step 850 is adapted such that computation of differencebetween the SUS and the polynomial at OGSP is not done. That is, method800 may be adapted to skip step 850. In various embodiments, step 860 isadapted such that a check flag indicative of a good fit is set to “yes”such that method 800 moves from step 860 to step 890. That is, method800 is adapted to not iterate and increase the polynomial orders.

It can be appreciated by one skilled in the art that method 800 issimilar in process and computation to method 500, except that, oneparameter polynomials are generated in method 500 while two parameterpolynomials are generated in method 800.

FIG. 9B graphically depicts an embodiment where K1=3 and K2=2. It isnoted that the grid points are centered at the CP and are labeled A_(ij)for i=1, 2, 3 and j=1, 2. The process for generating the two parameterpolynomial for this embodiment is the same as for the embodiment whereK1=2 and K2=2.

It is noted that the grid points may also be referred to as modelgeneration points.

Embodiments of step 220 indicate development of 3FA, 4FA, . . . , NFAmodels. These processes are indicated by steps 360-380 of the variousembodiments of step 220 discussed with respect to FIGS. 3A-3E. Theprocess for building the model has been described in detail for 1FA and2FA processes. It is contemplated that polynomials of three parameterterms, four parameter terms and so on may readily be generated asrequired. For 3FA, 4FA, . . . , NFA and the like, model generationpoints, OGSP, grid points, and RSP will be in the space of increasingdimension respectively. For example, for 3 parameter terms, modelgeneration points and the like will be points in the 3-dimensional(N-dimensional where N is 3) space defined by the APLs of the threerespective input parameters. For 4 parameter terms, model generationpoints and the like will be points in the 4-dimensional hyper space(N-dimensional where N is 4) defined by the APLs of the four respectiveinput parameters. Similarly, for N parameter terms, model generationpoints and the like will be points in the N-dimensional space defined bythe APLs of the N respective input parameters.

Step 360 of methods 300A-300E follows the same processes as those forsteps 320 and 340 except that the methods 700A-700B are adapted to buildapproximation models for higher order terms. For example, for 3FAanalysis, step 740 is adapted to one additional dimension (e.g.,3-dimensions instead of 2-dimensions) and interaction analysis isperformed for parameter triplets instead of parameter pairs. For 3FAanalysis, step 750 is adapted to select the highest ranked parametertriplet instead of a parameter pair. At step 760, a polynomial withthree parameter terms is generated instead of a polynomial with twoparameter terms. For 3FA analysis, step 770 is adapted so the model isupdated with the polynomial with three parameter terms for the selectedparameter triplet and the parameter triplet is removed from thecandidate parameter triplet list.

The polynomial generation process, discussed in detail for two parameterterms in method 800 is adapted to build approximation models for higherorder terms. For example, for 3FA analysis, step 810 is adapted to setup a three dimensional grid in the space defined by the APLs for theselected parameter triplet instead of a two dimensional grid. Step 830is adapted to generate a 3-parameter polynomial for the selected inputparameter triplet instead of a 2 parameter polynomial for the selectedinput parameter pair. Step 840 is adapted so the OGSP are in the 3dimensional space instead of the 2 dimensional plane. Step 890 may beadapted to remove the constant term, the 1-parameter terms, and the2-parameter terms as opposed to removing the constant term and the1-parameter terms. That is, when building a polynomial for a 3-tuplet ofinput parameters, step 890 may be adapted such that some or all of thosepolynomial terms that are missing one, two or all three input parametersmay be removed. For example, in various embodiments of building apolynomial for a N-tuplet of input parameters, some or all of thosepolynomial terms that do not use all of the N input parameters from theselected N-tuplet parameters may be removed from further considerationand/or inclusion in the NFA model being improved. For NFA analysis instep 360, model fit methods 700A-700B and polynomial generation method800 are appropriately adapted to the N level dimension. If N is equal to5, then the model generation points, the OGSP, the number of parametersin the polynomial term, and so on for the methods are appropriatelyadapted to the 5^(th) dimension. Those skilled in the art will readilybe able to follow the methods describe herein to any N^(th) dimensionmodel fit and polynomial generation.

In various embodiments, the terminology N-tuplet of input parameters,where N is an integer greater than 0, is used to refer to a group of Ninput parameters. When N=1, N-tuplet it refers to a single inputparameter; when N=2, N-tuplet refers to a pair of input parameters; whenN=3, N-tuplet refers to a triplet of input parameters; and so on.

In various embodiments, the terminology N-dimensional space, where N isan integer greater than 0, is used while working with NFA process. WhenN=1, N-dimensional space stands for a straight line. When N=2,N-dimensional space stands for a plane. When N=3, N-dimensional spacerefers to a 3-dimensional space. When N>3, N-dimensional space standsfor an N-dimensional hyper space.

In various embodiments, in methods 400A-400B described herein withrespect to FIGS. 4A and 4B, the approximation model at steps 420, 430,440, 450, 460, 470, 480, and 485 may be referred to as the currentapproximation model. In various embodiments, the approximation model atsteps 490 may be referred to as the improved approximation model.

In various embodiments, in methods 700A-700B described herein withrespect to FIGS. 7A and 7B, the approximation model at steps 720, 730,740, 750, 760, 770, 780, and 785 may be referred to as the currentapproximation model. In various embodiments, the approximation model atsteps 790 may be referred to as the improved approximation model.

FIG. 10 depicts a flow diagram of a method of processing two outputparameters according to various embodiments. In particular, the method1000 of FIG. 10 contemplates a User or System specifying, e.g., twooutput parameters within the context of the RACE mechanisms. Generallyspeaking, if the User supplies two output parameters, step 220 isexecuted for each output parameter. Some data exchanges and computationssuch as SUS evaluation information, stress points SP, and other data maybe shared and/or coordinated between the step 220-1 execution for outputparameter 1 and step 220-2 execution for output parameter 2. Thisinformation can be used to reduce data required, improve accuracy, orfor other purposes. If a User or System specifies four outputparameters, step 220 may be executed for each of the four outputparameters.

For each output parameter, User or System may have specified differentgoodness of fit criteria or Constraints. It is envisioned that for oneoutput, the method 1000 may proceed to a particular analysis level whilefor another output parameter the method 1000 may proceed to a differentanalysis level. For example, the method 1000 may proceed to a 2FAprocess for one output parameter while for another output parameter themethod 1000 may proceed to a 4FA process. This may be due to the factthat the goodness of fit criteria is achieved earlier for one outputparameter or it may be due to another one of the logical checks in themethods described.

FIG. 11 graphically depicts a flow diagram of method 1100 of generatingevaluation points, generating model generation points, generating stresspoints, generating input vectors, and evaluating a SUS, SCMM, CSCMM, NFAmodels, and the like according to one embodiment and suitable for use inimplementing step 310 and other steps in various figures. Various othermethods and mechanisms for generating vectors, selecting stress points,evaluating SUS, SCMM, CSCMM, NFA models, and so on are described herein.

At step 1110, method 1100 (e.g. via the TVE) generates evaluation pointsor stress points such as model generation points, APL points, gridpoints, SP, ASP, RSP, OGSP and the like using one or moremethods/mechanisms, such as orthogonal array design, fractionalfactorial design, full factorial design, space filling design and/orother design mechanisms. Generally speaking, these methods/mechanismsare adapted to provide input vectors comprising input parameters withinone or more regions of interest.

At step 1120, method 1100 (e.g. via the TVE) generates a plurality ofinput vectors corresponding to the points generated in step 1110 withinone or more regions of interest.

At step 1130, the SUS, current SUS-characterizing mathematical modelCSCMM, NFA model, and/or SCMM is evaluated at each of the input vectorswithin one or more regions of interest.

Generally speaking, SUS and SUS approximations are evaluated at thegenerated input vectors (i.e., sets of input parameters) within the oneor more regions of interest to provide thereby corresponding outputvectors (i.e., sets of outputs).

FIG. 12 describes a method of post-processing SUS characterizinginformation according to one embodiment. That is, the method 1200 ofFIG. 12 illustrates a use of SUS characterizing information such asdescribed above with respect to the various figures. Otherpost-processing methodologies using the information described herein arealso contemplated by the inventors.

At step 1210, SUS characterizing information is generated according toany of the techniques described above with respect to the variousfigures to provide thereby SUS characterizing information comprising,illustratively, an SUS-characterizing mathematical model in an equationformat for each output parameter, various input parameters anddistributions of input parameter values, ranges/regions of interest ofvarious input parameters and other information.

The SUS characterizing information may be generated in real time andcommunicated to one or more post-processing functional elements/modules.The SUS model information may have been previously generated and stored(e.g. Model Data 122-MD, Engine Data 122-ED, and the like), in whichcase the SUS characterizing information is retrieved from theappropriate storage location/module.

At step 1220, the generated/retrieved SUS characterizing information isprovided to one or more post-processing functional elements/modules(e.g., activation server 107 or another external data storage and/orprocessing device) for post-processing, storage for laterpost-processing and/or storage.

At step 1230, the activation server 107 and/or another external datastorage and/or processing device processes the SUS characterizinginformation to responsively generate summary data or other useful data,such as by using the equation(s) and associated information. Summarydata can include minima, maxima, mean, standard deviation, variance,inflection points, zeroes of the equation, transformations of theequation, percentile data, confidence intervals, integrals of equations,derivatives of equations, factor effects, interaction effects, and otherdata of interest.

At step 1240, SUS-related summary data may be output from the one ormore post-processing functional elements/modules for communication tothe SUS, another system, one or more end users and the like.Additionally, algorithms and decision processes of the SUS and/oranother system may be updated using the summary data. Additionally, theinput parameters, regions of interest or other SUS characterizinginformation may be adapted and the methodology rerun. This adaptationmay be responsive to new information of any type, changes in the SUS,user requirements/demands or other information.

Various embodiments contemplate multiple iterations within and/or acrossone or more of a plurality of approximations models (e.g., 1FA, 2FA, . .. , NFA). Further, various embodiments contemplate that theapproximation models iterated thereby may be repeated and/or iterated ina non-sequential manner (e.g., 1FA, then 2FA then 1FA again, then 3FAand so on). That is, while a NFA model is normally determined prior to a(N+1)FA model, the various steps to provide any of the models may berepeated and, more particularly, may be repeated in any order.

FIG. 13 depicts a high-level block diagram of a computing device, suchas a processor in a server or other computing device, suitable for usein performing functions described herein such as those associated withthe various elements described herein with respect to the figures.

As depicted in FIG. 13, computing device 1300 includes a processorelement 1303 (e.g., a central processing unit (CPU) and/or othersuitable processor(s)), a memory 1304 (e.g., random access memory (RAM),read only memory (ROM), and the like), a cooperating module/process1305, and various input/output devices 1306 (e.g., a user input device(such as a keyboard, a keypad, a mouse, and the like), a user outputdevice (such as a display, a speaker, and the like), an input port, anoutput port, a receiver, a transmitter, and storage devices (e.g., apersistent solid state drive, a hard disk drive, a compact disk drive,and the like)).

It will be appreciated that the functions depicted and described hereinmay be implemented in hardware and/or in a combination of software andhardware, e.g., using a general purpose computer, one or moreapplication specific integrated circuits (ASIC), and/or any otherhardware equivalents. In one embodiment, the cooperating process 1305can be loaded into memory 1304 and executed by processor 1303 toimplement the functions as discussed herein. Thus, cooperating process1305 (including associated data structures) can be stored on a computerreadable storage medium, e.g., RAM memory, magnetic or optical drive ordiskette, and the like.

It will be appreciated that computing device 1300 depicted in FIG. 13provides a general architecture and functionality suitable forimplementing functional elements described herein or portions of thefunctional elements described herein.

It is contemplated that some of the steps discussed herein may beimplemented within hardware, for example, as circuitry that cooperateswith the processor to perform various method steps. Portions of thefunctions/elements described herein may be implemented as a computerprogram product wherein computer instructions, when processed by acomputing device, adapt the operation of the computing device such thatthe methods and/or techniques described herein are invoked or otherwiseprovided. Instructions for invoking the inventive methods may be storedin tangible and non-transitory computer readable medium such as fixed orremovable media or memory, and/or stored within a memory within acomputing device operating according to the instructions.

Various modifications may be made to the systems, methods, apparatus,mechanisms, techniques and portions thereof described herein withrespect to the various figures, such modifications being contemplated asbeing within the scope of the invention. For example, while a specificorder of steps or arrangement of functional elements is presented in thevarious embodiments described herein, various other orders/arrangementsof steps or functional elements may be utilized within the context ofthe various embodiments. Further, while modifications to embodiments maybe discussed individually, various embodiments may use multiplemodifications contemporaneously or in sequence, compound modificationsand the like.

The various embodiments discussed above provide significant advantageswithin context of numerous types of analysis or modelingapplications/uses, including weather prediction, machine/engineperformance analysis, voice compression/evaluation, videocompression/evaluation, automobile, aircraft or other machine/systemperformance under various operating conditions, product manufacturingyield, environmental and other resource management and so on.

Although various embodiments which incorporate the teachings of thepresent invention have been shown and described in detail herein, thoseskilled in the art can readily devise many other varied embodiments thatstill incorporate these teachings. Thus, while the foregoing is directedto various embodiments of the present invention, other and furtherembodiments of the invention may be devised without departing from thebasic scope thereof. As such, the appropriate scope of the invention isto be determined according to the claims.

What is claimed is:
 1. An apparatus for modeling a System Under Study(SUS), the SUS exhibiting at least one output adapted in response to aplurality of input parameters, the apparatus comprising a processorconfigured to: generate an improved approximation model for said SUScomprising a summation of an initial approximation model, a constantterm, and a plurality of polynomials, each polynomial being associatedwith a respective N-tuplet of input parameters for greater than zerointeger values of N, wherein polynomials included within said improvedapproximation model are generated by iteratively performing thefollowing steps until a sufficiently accurate fit is achieved: rankordering the N-tuplets of input parameters; selecting a highest rankingN-tuplet of input parameters from among the remaining N-tuplets of inputparameters to be processed; fitting a polynomial involving inputparameters in the selected N-tuplet of input parameters; updating thecurrent approximation model by adding the fitted polynomial; adjustingthe constant term as needed to match the evaluations of the SUS and thecurrent approximation model at a point; and evaluating fit of the saidcurrent approximation model to SUS; wherein data representative of theimproved approximation model of said SUS is stored in memory.
 2. Theapparatus in claim 1, wherein fitting a polynomial involving inputparameters in a selected N-tuplet of input parameters comprises:iteratively performing the following steps until a sufficiently accuratefit is achieved: selecting a plurality of model generation points in theN-dimensional space defined by axis parallel lines of the selectedN-tuplet of input parameters, where model generation points areincremented by one or more for subsequent iterations in the saidN-dimensional space; evaluating the SUS at the model generation points;fitting a polynomial involving input parameters from the said N-tupletof input parameters to the SUS evaluation; and evaluating fit of saidpolynomial to SUS; and removing polynomial terms that do not involve allinput parameters from the said N-tuplet of input parameters.
 3. Theapparatus of claim 1, wherein rank ordering is determined according to adifference between evaluations of SUS and the current approximationmodel at ranking stress points generated using at least one oforthogonal array design, fractional factorial design, full factorialdesign, design of experiments and space filling designs.
 4. Theapparatus in claim 1, wherein rank ordering is determined according toat least one of importance, presumed importance, random order, pastdata, alphabetical order and order of entry.
 5. The apparatus of claim4, wherein importance of input parameters is determined by main effectsanalysis of the difference between SUS evaluations and currentapproximation model evaluations at ranking stress points.
 6. Theapparatus of claim 4, wherein importance of N-tuplets of inputparameters is determined by N-parameter interaction effects analysis ofthe difference between evaluations of SUS and the current approximationmodel at ranking stress points.
 7. The apparatus of claim 1, whereinfitting a polynomial involving input parameters in the selected N-tupletof input parameters is performed using at least one of orthogonalpolynomials, matrix inversion, Lagrange polynomials and solvingsimultaneous equations.
 8. The apparatus of claim 1, wherein sufficiencyof accuracy is determined by goodness of fit criteria comprising atleast one of a maximum absolute difference, a sum of absolutedifferences, a mean square of selected top absolute differences, and amean square error.
 9. The apparatus of claim 1, wherein saidsufficiently accurate fit is achieved in response to reaching at leastone of a dimensional limit, a cost constraint, a time constraint and aresource constraint.
 10. The apparatus of claim 1, wherein the initialapproximation model is equal to one of the following: zero, a userspecified model, a system specified model and a previously improvedapproximation model.
 11. The apparatus of claim 1, wherein the fit isdetermined by generating one or more stress points and evaluating adifference between evaluations of the current approximation model andthe SUS at the said stress points according to at least one goodness offit criteria comprising an absolute difference, a sum of absolutedifferences, a mean square of selected top absolute differences, and amean square error.
 12. The apparatus of claim 11, wherein stress pointsare generated using at least one of orthogonal arrays, fractionalfactorial designs, full factorial designs, design of experiments andspace filling designs.
 13. The apparatus of claim 11, wherein one ormore stress points generated in one iteration are used in at least onesubsequent iteration.
 14. The apparatus of claim 11, wherein one or morestress points are generated for each iteration.
 15. The apparatus ofclaim 1, further comprising: generating summary statistics associatedwith the improved approximation model of said SUS; and storing datarepresentative of generated summary statistics in memory.
 16. Theapparatus of claim 15, wherein said summary statistics comprise at leastone of mean, standard deviation, maxima, minima, inflection points,confidence intervals, probability percentiles, zeros of an equation,integrals associated with one or more input parameters and derivativesassociated with one or more input parameters.
 17. The apparatus of claim1, further comprising: post-processing said improved approximation modelof said SUS according to at least one of an equation, a distribution anda region of interest of input parameters to generate thereby decisionrules; and storing data representative of the decision rules in memory.18. The apparatus of claim 1, wherein rank ordering of input parametersperformed only in a first iteration.
 19. The apparatus of claim 1,wherein a sufficiency of accuracy parameter is adapted so that aniteration is performed until polynomials for all of a specified set ofN-tuplets of input parameters have been included in the currentapproximation model.
 20. The apparatus of claim 19, wherein evaluationof fit of the current approximation model to the SUS occurs only afterpolynomials for all of a specified set of N-tuplets of input parametershave been included in the current approximation model.
 21. The apparatusof claim 1, wherein fitting a polynomial involving input parameters inthe selected N-tuplet of input parameters comprises constructing apolynomial with an origin proximate the center point.
 22. The apparatusof claim 1, wherein fitting a polynomial involving input parameters inthe selected N-tuplet of input parameters is accomplished by performingthe steps of: selecting a plurality of model generation points in theN-dimensional space defined by axis parallel lines of the selectedN-tuplet of input parameters; evaluating the SUS at the model generationpoints; fitting a polynomial involving input parameters from the saidN-tuplet of input parameters to the SUS evaluation; and removingpolynomial terms that do not involve all input parameters from the saidN-tuplet of input parameters.
 23. The apparatus of claim 2, wherein thefit is determined by: generating one or more stress points in theN-dimensional space defined by the axis parallel lines of the inputparameters in the N-tuplet of input parameters that are distinct fromthe model generation points; and evaluating a difference betweenevaluations of the said polynomial and the SUS at the said stress pointsaccording to at least one goodness of fit criteria comprising absolutedifference, sum of absolute differences, mean square of selected topabsolute differences and mean square error.
 24. The apparatus of claim23, wherein stress points are generated using at least one of orthogonalarrays, fractional factorial designs, full factorial designs, design ofexperiments and space filling designs.
 25. The apparatus in claim 1,wherein fitting a polynomial involving input parameters in the selectedN-tuplet of input parameters comprises: iteratively by performing thefollowing steps until a sufficiently accurate fit is achieved: selectinga plurality of model generation points in or close to the N-dimensionalspace defined by axis parallel lines of the selected N-tuplet of inputparameters, where model generation points are incremented by at leastone subsequent iteration within approximately the said N-dimensionalspace; evaluating the SUS at the model generation points; fitting apolynomial involving input parameters from the said N-tuplet of inputparameters to the SUS evaluation; and evaluating fit of said polynomialto SUS; and removing polynomial terms that do not involve all parametersfrom the said N-tuplet of input parameters.
 26. The apparatus in claim1, wherein fitting a polynomial involving input parameters in theselected N-tuplet of input parameters comprises iteratively performingthe following steps until a sufficiently accurate fit is achieved:selecting a plurality of model generation points in the N-dimensionalspace defined by axis parallel lines of the selected N-tuplet of inputparameters, where model generation points are incremented by at leastone for subsequent iterations in the said N-dimensional space;evaluating the SUS at the model generation points; fitting a polynomialinvolving input parameters from the said N-tuplet of input parameters tothe SUS evaluation; and evaluating fit of said polynomial to SUS. 27.The apparatus of claim 2, wherein fitting a polynomial involving inputparameters from the said N-tuplet of input parameters to the SUSevaluation is performed using at least one of orthogonal polynomials,matrix inversion, Lagrange polynomials and solving simultaneousequations.
 28. The apparatus of claim 2, wherein sufficiency of accuracyis determined by goodness of fit criteria comprising at least onegoodness of fit criterion comprising an absolute difference, a sum ofabsolute differences, a mean square of selected top absolute differencesand a mean square error.
 29. An apparatus for modeling a System UnderStudy (SUS), the SUS exhibiting at least one output adapted in responseto a plurality of input parameters, the apparatus comprising a processorconfigured to: generate an improved approximation model for said SUScomprising a summation of an initial approximation model, a constantterm, and a plurality of complex function elements, each complexfunction element being associated with a respective N-tuplet of inputparameters for greater than zero integer values of N, wherein complexfunction elements included within said improved approximation model aregenerated by iteratively performing the following steps until asufficiently accurate fit is achieved: rank ordering the N-tuplets ofinput parameters; selecting a highest ranking N-tuplet of inputparameters from among the remaining N-tuplets of input parameters to beprocessed; fitting a complex function element involving input parametersin the selected N-tuplet of input parameters; updating the currentapproximation model by adding the fitted complex function element;adjusting the constant term as needed to match the evaluations of theSUS and the current approximation model at a point; and evaluating fitof the said current approximation model to SUS; wherein datarepresentative of the improved approximation model of said SUS is storedin memory.
 30. The apparatus of claim 29, wherein the complex functionelements are associated with one of a polynomial function, a waveletfunction, a Fourier series and a Walsh function.
 31. A tangible andnon-transient computer readable storage medium storing instructionswhich, when executed by a computer, adapt the operation of the computerto provide a method of modeling a System Under Study (SUS), the SUSexhibiting at least one output adapted in response to a plurality ofinput parameters, the method comprising: generating an improvedapproximation model for said SUS comprising a summation of an initialapproximation model, a constant term, and a plurality of polynomials,each polynomial being associated with a respective N-tuplet of inputparameters for greater than zero integer values of N, whereinpolynomials included within said improved approximation model aregenerated by iteratively performing the following steps until asufficiently accurate fit is achieved: rank ordering the N-tuplets ofinput parameters; selecting a highest ranking N-tuplet of inputparameters from among the remaining N-tuplets of input parameters to beprocessed; fitting a polynomial involving input parameters in theselected N-tuplet of input parameters; updating the currentapproximation model by adding the fitted polynomial; adjusting theconstant term as needed to match the evaluations of the SUS and thecurrent approximation model at a point; and evaluating fit of the saidcurrent approximation model to SUS; wherein data representative of theimproved approximation model of said SUS is stored in memory.
 32. Acomputer program product wherein computer instructions, when executed bya processor in a computing device, adapt the operation of the computingdevice to provide a method of modeling a System Under Study (SUS), theSUS exhibiting at least one output adapted in response to a plurality ofinput parameters, the method comprising: generating an improvedapproximation model for said SUS comprising a summation of an initialapproximation model, a constant term, and a plurality of polynomials,each polynomial being associated with a respective N-tuplet of inputparameters for greater than zero integer values of N, whereinpolynomials included within said improved approximation model aregenerated by iteratively performing the following steps until asufficiently accurate fit is achieved: rank ordering the N-tuplets ofinput parameters; selecting a highest ranking N-tuplet of inputparameters from among the remaining N-tuplets of input parameters to beprocessed; fitting a polynomial involving input parameters in theselected N-tuplet of input parameters; updating the currentapproximation model by adding the fitted polynomial; adjusting theconstant term as needed to match the evaluations of the SUS and thecurrent approximation model at a point; and evaluating fit of the saidcurrent approximation model to SUS; wherein data representative of theimproved approximation model of said SUS is stored in memory.